How to set PYTHONPATH in Visual Studio Code?

[aupres]

Hello!
Belows are my dev environment,

OS : Windows 11
python : Anaconda3 
Apache Spark : 3.4.1
IDE : Visual Studio Code 1.18.1

And I try to integrate apache spark pyspark into VS code. So I set the python path into settings.json file of vs code.

"python.defaultInterpreterPath": "C:\\Anaconda3\\python.exe",
"python.condaPath": "C:\\Anaconda3\\Scripts\\conda.exe",
"terminal.integrated.env.windows": {
	"PYTHONPATH": "C:/spark-3.4.1-bin-hadoop3/python;C:/spark-3.4.1-bin-hadoop3/python/pyspark;C:/spark-3.4.1-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip;C:/spark-3.4.1-bin-hadoop3/python/lib/pyspark.zip"
},
"python.autoComplete.extraPaths": [
        "C:\\spark-3.4.1-bin-hadoop3\\python",
        "C:\\spark-3.4.1-bin-hadoop3\\python\\pyspark",
        "C:\\spark-3.4.1-bin-hadoop3\\python\\lib\\py4j-0.10.9.7-src.zip",
        "C:\\spark-3.4.1-bin-hadoop3\\python\\lib\\pyspark.zip"
],
"python.analysis.extraPaths": [
        "C:\\spark-3.4.1-bin-hadoop3\\python",
        "C:\\spark-3.4.1-bin-hadoop3\\python\\pyspark",
        "C:\\spark-3.4.1-bin-hadoop3\\python\\lib\\py4j-0.10.9.7-src.zip",
        "C:\\spark-3.4.1-bin-hadoop3\\python\\lib\\pyspark.zip"
]

The pyspark works without errors with these configuration. But the issue happens when I import pandas of python default module.

import pandas as pd

This simple and basic expression throws the errors like below,

Error:
import pandas as pd
  File "C:\spark-3.4.1-bin-hadoop3\python\pyspark\pandas\__init__.py", line 29, in <module>
    from pyspark.pandas.missing.general_functions import MissingPandasLikeGeneralFunctions
  File "C:\spark-3.4.1-bin-hadoop3\python\pyspark\pandas\__init__.py", line 34, in <module>
    require_minimum_pandas_version()
  File "C:\spark-3.4.1-bin-hadoop3\python\pyspark\sql\pandas\utils.py", line 37, in require_minimum_pandas_version
    if LooseVersion(pandas.__version__) < LooseVersion(minimum_pandas_version):
                    ^^^^^^^^^^^^^^^^^^
AttributeError: partially initialized module 'pandas' has no attribute '__version__' (most likely due to a circular import)

As you see, the imported pandas module is not python module, but pyspark.pandas module. So the code brings the error. I set the python default interpreter to anaconda3 at the top line of the settings.json file. But it still brings the errors. Any reply will be thanksful.

[snippsat]

Do not mess with Path setting in VS Code,click down in right corner and choice right environment(Python interpreter).
See that a conda environment look like this 3.10.1(tom_env: conda).

So this will work if eg start a blank eg VSCodium,with no settings.json setup.
Also from command line always activate a environment using Anaconda or MiniConda,eg (base) or should make your own.
Here i activate my own (tom_env) and start VSCode and VSCodium.

[aupres]

Thanks for your reply. In my vs code, I have only one python interpreter - conda base. And I always activate the conda. But the same error is thrown.
   

[snippsat]

Delete all you have shown in first in settings.json .
in your error messgage it load Pandas from:

Error:
C:\spark-3.4.1-bin-hadoop3\python\pyspark\pandas\__init__.py 

This is wrong for you with (base) Anaconda it should load from:

C:\Anaconda\Lib\site-packages\pandas\... 

If i test eg with a settings.json with no config setup at all.

{	
	"workbench.iconTheme": "simple-icons",
	"workbench.productIconTheme": "fluent-icons",
	"window.zoomLevel": 1,
	"workbench.colorTheme": "Visual Studio Dark",
}

Test code

import sys
import pandas as pd

print(sys.executable)
print(pd.__file__)

So running code over eg in 3.10.1(tom_env: conda).

Output:
G:\miniconda3\envs\tom_env\python.exe
G:\miniconda3\envs\tom_env\lib\site-packages\pandas\__init__.py

See that it ponit to same root folders.
If change to eg Python 3.11.3 from python.org.

Output:
C:\Python311\python.exe
C:\Python311\Lib\site-packages\pandas\__init__.py

See the same pattern,ovet to eg 3.10.8(sci_env: conda).

Output:
G:\miniconda3\envs\sci_env\python.exe
G:\miniconda3\envs\sci_env\lib\site-packages\pandas\__init__.py

Pandas do of course need to be installed in all envioment,there is no sharing or loading from other Paths,then it will fail.

[aupres]

I appreciate your reply. One of my project environments is the integration of Apache Spark. So I have to import pyspark module. Lots of python files include pyspark module. For example,

from pyspark.sql import SparkSession

Here is my dilemma. If I erase the PYTHONPATH from the settings.json file,

Error:
ModuleNotFoundError: No module named 'pyspark'

is thrown. Then I insert the PYTHONPATH, the above error occurs. I make every efforts. For exmaple, In launch.json file, I type in the following codes,

{
	"type": "python",
    "name": "Python: Current File",
    "request": "launch",
    "program": "${file}",
    "console": "integratedTerminal",
    "cwd": "${fileDirname}",
    "env": {
                "PYTHONPATH": "C:/spark-3.4.1-bin-hadoop3/python;C:/spark-3.4.1-bin-hadoop3/python/pyspark;C:/spark-3.4.1-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip;C:/spark-3.4.1-bin-hadoop3/python/lib/pyspark.zip"
            }
}

But it does not works at all. Desperately need your advice.

[snippsat]

But it does not works at all. Desperately need your advice.

Do not give path VS Code.
This should work fom commandline before using any Editors.
If i do test install of pyspark using eg conda Installation

conda install -c conda-forge pyspark

I have conda-forge on as default.

# Activate environment  
G:\div_code\egg\ping
λ G:\miniconda3\Scripts\activate.bat tom_env
# Install pyspark
(tom_env) G:\div_code\egg\ping
λ conda install pyspark
Retrieving notices: ...working... done
.....

Test that it work.

(tom_env) G:\div_code\egg\ping
λ ptpython
>>> import pandas as pd
>>> pd.__file__
'G:\\miniconda3\\envs\\tom_env\\lib\\site-packages\\pandas\\__init__.py'

>>> from pyspark.sql import SparkSession
>>> spark = SparkSession.builder.getOrCreate()

# Testing pyspark
>>> from datetime import datetime, date
... import pandas as pd
... from pyspark.sql import Row
...
... df = spark.createDataFrame([
...     Row(a=1, b=2., c='string1', d=date(2000, 1, 1), e=datetime(2000, 1, 1, 12, 0)),
...     Row(a=2, b=3., c='string2', d=date(2000, 2, 1), e=datetime(2000, 1, 2, 12, 0)),
...     Row(a=4, b=5., c='string3', d=date(2000, 3, 1), e=datetime(2000, 1, 3, 12, 0))
... ])
>>> df
DataFrame[a: bigint, b: double, c: string, d: date, e: timestamp]

>>> df.columns
['a', 'b', 'c', 'd', 'e']

So now all work from commandline,without any Editors or setup in them.

For Editors i have to set this,so set PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON.
This is a OS environment setup,and not a setup in Editors.

(tom_env) G:\div_code\egg\ping
λ set PYSPARK_PYTHON=G:\miniconda3\envs\tom_env\python.exe

(tom_env) G:\div_code\egg\ping
λ set PYSPARK_DRIVER_PYTHON=G:\miniconda3\envs\tom_env\python.exe 

Then this work in VS Code with no setup in Editor,other than activate environment.

from datetime import datetime, date
import pandas as pd
from pyspark.sql import Row
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

df = spark.createDataFrame([
    Row(a=1, b=2., c='string1', d=date(2000, 1, 1), e=datetime(2000, 1, 1, 12, 0)),
    Row(a=2, b=3., c='string2', d=date(2000, 2, 1), e=datetime(2000, 1, 2, 12, 0)),
    Row(a=4, b=5., c='string3', d=date(2000, 3, 1), e=datetime(2000, 1, 3, 12, 0))
])

print(df) 

---

If you messed (base) environment just make a new one,this is important point of using Anaconda.
Do not try to fix a broken environment,just make new one.
Example,this also install new Python 3.10.8 version and notebook i a most have with jupyterlab.

conda create --name my_env -c conda-forge pyspark jupyterlab python=3.10.8