I'm currently trying to learn more about the layers API of tensorflow, for this I'm trying the cloud-ml samples (census: https://github.com/GoogleCloudPlatform/cloudml-samples/tree/master/census).
When I launch the script on my Windows computer (Windows-10, run in local, not distributed, CPU mode), I get the following error:
File "\*\Anaconda3\lib\site-packages\tensorflow\contrib\layers\python\layers\feature_column.py", line 1652, in insert_transformed_feature name="bucketize")
File "\*\Anaconda3\lib\site-packages\tensorflow\contrib\layers\python\ops\bucketization_op.py", line 48, in bucketize
return _bucketization_op.bucketize(input_tensor, boundaries, name=name)
AttributeError: 'NoneType' object has no attribute 'bucketize'
In the code of tensorflow (I used version 1.0.0, and upgraded it to 1.0.1 with the same error), I saw in the file tensorflow\contrib\layers\python\ops\bucketization_op.py that the op was loaded from native code:
_bucketization_op = loader.load_op_library(
resource_loader.get_path_to_datafile("_bucketization_op.so"))
At this point I actually have two questions:
Am I wrong to think that this is only valid on Linux, or the .dll might have been renamed .so to keep a coherent Python code? If there's such a renaming, can someone tell me where I could find this file as a quick search into the folder gave no result for *.dll or *.so (I assume every native code is wrapped by SWIG inside the _pywrap_tensorflow.pyd)?
Does anyone have a clue of why this kind of error could happen?
TL;DR: These ops should now work in the current nightly build of TensorFlow. I've sent out a pull request to add support in the upcoming 1.1 release.
The explanation is a bit tortuous, but I'll attempt to lay out the key points.
In general, the tf.contrib libraries have limited support on Windows, often because they depend on platform-specific code that does not work (or has not historically worked) on Windows. Until very recently the tf.load_op_library() API did not work on Windows, but a recent pull request added support for it. Nightly builds for TensorFlow on Windows now include .dll files for some extension libraries, and the loader library includes code that converts the .so extension to .dll on Windows.
As a historical workaround for this problem, we statically linked every tf.contrib kernel into _pywrap_tensorflow.pyd, and made loader.load_op_library() fail silently if the extension was not present on Windows. However, there are two ways to get the generated Python wrapper functions for each op:
The more common way, which (e.g.) tf.contrib.tensor_forest uses is to generate the Python source at build time and include the generated code in the PIP package. This works fine on Windows.
The less common way, which bucketization_op.py uses is to generate the Python source at run time, and return a generated Python module from loader.load_op_library(). Since we made this fail silently and return None on Windows, calling _bucketization_op.bucketize() doesn't work.
Finally, due to operational concerns, we determined that it would be useful to switch between the static and dynamic linking of the tf.contrib kernels on all platforms, and the easiest way to do that would be to generate the wrapper code statically. A recent change (which alas just missed the branch for the 1.1 release) made the generation of wrapper code consistent across all of the tf.contrib libraries.
I hope this makes sense. As a result of all of these changes, if you upgrade to a nightly build of TensorFlow the problem should be fixed, and
hopefully we can merge the change into the 1.1 release as well!
Related
I'm trying to run a face detection model in Unity. It gets input from the webcam, then spits out a face. But trying to make this work with C# has been an absolute nightmare. And despite all my suffering, I still haven't been able to make it work!
If I could use python, I'd be able to get it done easily. So, obviously, I want to find a way to get a python script working in Unity. But IronPython is the only thing I've been able to find, and it's outdated.
I need either knowledge of how to make IronPython work in spite of being outdated, or some other method. Please.
Unfortunately, Unity at this time does not support Python. Although, there is an asset that you can use a bit of Python with. I am not sure what you can do with this asset but I know it could help a minimal amount:https://assetstore.unity.com/packages/tools/integration/python-interpreter-645
Quick Note: Most programming languages work about the same way. If you figure out the documentation and grammar/punctuation for C#/UnityC#, you should be off just fine.
I try to use python once on Unity and I found a few ways:
There is a package call "IronPython" where you can add a python file to your unity project and then call a function from C# to your python code, to do that you should follow this:
We already know that we can use python to use .net internal calls.
Now we may use the same to start a console that can accept a scripting language in Unity engine.
To do this we have to include certain dll files.
These dll files must be present in Assets>plugins
IronPython.dll
IronPython.Modules.dll
Microsoft.Scripting.Core.dll
Microsoft.Scripting.dll
Microsoft.Scripting.Debugging.dll
Microsoft.Scripting.ExtensionAttribute.dll
Microsoft.Dynamic.dll
Once the Plugins are in place.
Initiate the Cs code
PythonEngine engine = new PythonEngine();
engine.LoadAssembly(Assembly.GetAssembly(typeof(GameObject)));
engine.ExecuteFile("Test.py");
Where test.py is the python code.
Initiate python side:
import UnityEngine from UnityEngine
import *
Debug.Log("Hello world from IronPython!")
References:
https://github.com/cesardeazevedo/Unity3D-Python-Editor
http://techartsurvival.blogspot.in/2013/12/embedding-ironpython-in-unity-tech-art.html
IronPython in Unity3D
the issue with this way is that most of the python module are not supported.
2.the second way is to create a file like json that contain the data you want to send to the json and then create an output json that send the output from the python script, this way is very limited with what you can send because the data must be contain in your json.
the last way that work for me is to install the Nuget package and copy the script from python to c# line by line with the relevent module installed in Unity and it's work for me, but copy a long code can take time.
this is a reference to the package:
https://github.com/GlitchEnzo/NuGetForUnity
and then to install the relevent package you should press on NuGet → Manage NuGet Packages and the choose the relevent package(for me it was Numpy and it work grate).
hope it will help you
I don't know how recent it is but there is a Unity package for python available on unity 2019.3 and further versions.
Warning the first versions of this package can't use Python3.
You can see more for yourself by the following link.
https://docs.unity3d.com/Packages/com.unity.scripting.python#2.0/manual/index.html
I hope this may help you.
We are thrilled to announce that Python for Unity 4.0.0-exp.5 is now available!
4.0.0-exp.5 is a major upgrade from our last public release, and incorporates a large number of changes. In summary:
Based on Python 3.7; scripts based on Python 2.7 will need to be ported.
Users no longer need to install Python on their system.
In-process Python is no longer reinitialized when the Unity domain reloads.
Removed the out-of-process API. The PySide example now runs in-process and is much simpler.
Limited support for a virtual environment workflow via the ProjectSettings/requirements.txt file.
Many bug fixes.
Documentation for the Python for Unity package is available here, and the full changelog can be found here.
This is an experimental release, and thus is not visible in Package Manager. To install this package, open Package Manager, click the + at the top left and select Add package by name.... Enter com.unity.scripting.python as the name and and 4.0.0-exp.5 as the version and click Add. Alternatively, you may edit Packages/manifest.json and add "com.unity.scripting.python": "4.0.0-exp.5", to the list of dependencies, or edit the existing entry for Python for Unity to update the version.
Soursce: https://forum.unity.com/threads/python-for-unity-release-announcements.1084688/
Documentation: https://docs.unity3d.com/Packages/com.unity.scripting.python#4.0/manual/index.html
Unity not supported python, But you Can write Python Code and run it by Socket programing, Create Server with python and send data,in C# Connect to server and use data sended with python.
I am in the process of updating our build machine (from Windows Server 2008 to Windows 10), and we need to support both Python 2 and Python 3. We are using SWIG to generate Python bindings for our C++ code. As part of this we generate both a C++ only DLL (mylib_shared.dll, built directly using Clang) and a python-version-specific dll (_mylib.dll, built from swig).
The build is currently fine in Python 2.7, but for Python 3 when I try to import a function from _mylib.dll using the SWIG bindings, I get an error saying the shared dll is missing. This file is identical no matter which version of Python is running, and is definitely present, in the expected place and that directory is in the PATH variable (I've checked, very carefully). I have used Dependencies and dumpbin to check what it's looking for, the only difference I can see is that Python38.dll (which is obviously imported by the Python 3 version but not the Python 2 version) has a dependency on ws2_32.dll (present in C:\Windows\SysWOW64) that isn't a dependency for the equivalent Python 2 dll. That says it's missing api-ms-win-core-string-obsolete-l1-1-0.dll, but I think this is a red herring - that seems to be an implementation-dependent thing (see https://answers.microsoft.com/en-us/windows/forum/windows_10-files/missing-api-ms-win-core-dlls/d99d1368-0f92-43db-bbdb-7d080f1f96e9). I have checked that everything is being built for the correct architecture (32-bit) by loading the dlls with dumpbin.
So I'm stumped. I had a problem previously with mixed architectures causing a similar problem, but I resolved that and as I say, I'm now checking directly on the dlls being built, so I can't see it's that (also you'd expect it to be a problem for both versions of Python). The file is definitely there. I don't know what else would be causing Windows to claim it can't find the dll. Any help gratefully received!
I have encountered a rather funny situation: I work in a big scientific collaboration whose major software package is based on C++ and python (2.7.15 still). This collaboration also has multiple servers (SL6) to run the framework on. Since I joined the collaboration recently, I received instructions on how to set up the software and run it. All works perfectly on the server. Now, there are reasons not to connect to the server to do simple tasks or code development, instead it is preferrable to do these kind of things on your local laptop. Thus, I set up a virtual machine (docker) according to a recipe I received, installed a couple of things (fuse, cvmfs, docker images, etc.) and in this way managed to connect my MacBook (OSX 10.14.2) to the server where some of the libraries need to be sourced in order for the software to be compiled and run. And after 2h it does compile! So far so good..
Now comes the fun part: you run the software by executing a specific python script which is fed as argument another python script. Not funny yet. But somewhere in this big list of python scripts sourcing one another, there is a very simple task:
import logging
variable = logging.DEBUG
This is written inside a script that is called Logging.py. So the script and library only are different by the first letter: l or L. On the server, this runs perfectly smooth. On my local VM set up, I get the error
AttributeError: 'module' object has no attribute 'DEBUG'
I checked the python versions (which python) and the location of the logging library (print logging.__file__), and in both set ups I get the same result for both commands. So the same python version is run, and the same logging library is sourced but in one case there is a mix up with the name of the file that sources the library.
So I am wondering, if there is some "convention file" (like a .vimrc for vi) sourced somewhere where this issue could be resolved by setting some tolerance parameter to some other value...?
Thanks a lot for the help!
conni
as others have said, OSX treats names as case-insensitive by default, so the Python bundled logging module will be confused with your Logging.py file. I'd suggest the better fix would be to get the Logging.py file renamed, as this would improve compatibility of the code base. otherwise, you could create a "Case-sensitive" APFS file system using "Disk Utility"
if you go with creating a file system, I'd suggest not changing the root/system partition to case-sensitive as this will break various programs in subtle ways. you could either repartition your disk and create a case-sensitive filesystem, or create an "Image" (this might be slower, not sure how much) and work in there. Just make sure you pick the "APFS (Case-sensitive)" format when creating the filesystem!
I've downloaded the python 3.6.6 source from here...
https://www.python.org/downloads/release/python-366/
...and followed the instruction on how to build on Windows (run ../PCbuild/build.bat). Python compiles and seems to be working (funny and scary: while fetching externals, it actually downloads python-3.7.0 as a dependency... :/ ). However, it looks like the build is somehow 'in place', and the binaries end up in some sub-folder of the source (../PCbuild/amd64/python.exe). This means I'm left with source and compiled code mixed up instead of some clean/lean and deployable package.
can I somehow provide '--prefix=/target/build/path' to define a target location to build to, like I would on linux?
is there a way of removing all src files/folders and leave only the required files/folders (../lib, ../include, etc...).
Or in general, is there a way of making the build process more behave like on linux?
Thanks for your help,
Max
The build.bat from PCBuild is intended for developers, that is, for testing purposes. What you want is under \Tools\msi\buildrelease.bat. This creates a subdirectory under \PCBuild\ that places all msi, cab and exe files ready for later installation. According to the readme, there doesn't seem to be an option to pack all those files in a single .exe file, like all installers eventually do, but another option is under \Tools\msi\build.bat which does have an option for packing (namely build.bat --pack). "But", the readme does state that the buildrelease.bat should be used for an official release. The advantage of doing so is that Pyhton would be optimized using PGO to your own hardware. I am also trying to compile from source using this method but I am having an issue with a recurring error (and other ones):
PGO run did not succeed (no python36!*.pgc files) and there is no data to merge [E:\RepoGiT\3.6\PCbuild\pythoncore.vcxproj]
so, if you do go this route, and find this, or other errors, please send the bug report to python's bug tracker webpage. And better yet, if you find errors and their solution, please report back here!
I wanted to use OpenCV with Python, so I downloaded OpenCV for Windows and got a folder of ~3.7GB after decompression. What surprised me was that the only file I needed was cv2.pyd, which was so small (~11MB) comparing to the C builds (~674MB). I simply copied it to my Python lib-packages folder without adding anything to my PATH and it worked perfectly.
I don't know how Python binding works, and I thought it should call C/C++ implementations under the hood. However, cv2 did not seem to require any C/C++ library. It just looks like magic to me.
Most likely it has something to do with static linking and using all possible tricks found in "Reducing Executable Size" or "GCC x86 code size optimizations"
OpenCV uses cmake as build system, which provides "MinSizeRel" build type. It seems to auto-apply most of those tricks. Couldn't find any good documentation on that, hence: [citation needed]
(follows my original answer which didn't quite address the actual question)
More convenient way to get opencv for python may be to download it from: http://www.lfd.uci.edu/~gohlke/pythonlibs/#opencv
After running installer you'll find cv2.pyd in c:\python27\lib\site-packages
As far as we are concerned .pyd file is same as .dll: http://docs.python.org/2/faq/windows.html#is-a-pyd-file-the-same-as-a-dll
Which means that we can use Dependency Walker to look into it. This is what we see:
This picture means that cv2.pyd is dynamically linked against opencv libraries which contain actual functionality. These take around ~45MB of disk space.