We Keep Coding

MongoDB Python: Unable to remove a field that has a . in it

I'm using pymongo and I'm unable to remove a field that has a "." in the field name. I've tested this same code with other fields that do not have a period and it works without an issue.
db.city.update_many({}, {'$unset': {'AverageDailyPM2.5':1}})
I'm pretty certain I'm on the latest versions of everything including MongoDB 4.4.9.

That is one of the reasons it is not recommended to use "." in a field name.
If you use the pipeline form of update, you can convert the object to an array with $objectToArray so the fields become data, remove the desired field from the array with $filter, convert back to an object with $arrayToObject, and then use $replaceRoot to make that the final document.

Using Flask's MongoEngine. Error with datetime field

I have a model with the a datetime field
user_since = db.DateTimeField()
When I try to insert a new object of the model into mongo, There is no error. But the write does not succeed.
I printed the object from to_json() and tried to insert it with mongo shell, I get the following error.
field names cannot start with $ [$date] at src/mongo/shell/collection.js:L147
the to_json had this field.
"user_since": {"$date": 1392205572989}
I can't seem to find any pointers on how to solve this.
What is causing the write to fail?
How can I make mongoengine to throw error in case of write failure.? Or at least find out what the error was?
Thanks.
Update:
As I found later the real problem is not the datetime field. The details of the problem are in this question MongoEngine Document Object made using from_json doesn't save

With MongoDB you cannot have '$' at the beginning of the field name.
MongoDB Limits and Thresholds - Restrictions on Field Names:
Field names cannot contain dots (i.e. .), dollar signs (i.e. $), or
null characters. See Dollar Sign Operator Escaping for an alternate
approach.
Try to name it differently.
edit:
According to MongoEngine documentation, you could pass db_name parameter:
db_field – The database field to store this field in (defaults to the
name of the field)

How to serialize and deserialize Django ORM query (not queryset)?

My use case is that I need to store queries in DB and retrieve them from time to time and evaluate. Thats needed for mailing-app where every user can subscribe to a web-site content selected by individually customized query.
Most basic solution is to store raw SQL and use it with RawQuerySet. But I wonder is there better solutions?

At first glance, it is really dangerous to hand out query building job to others, since they can do anything (even delete all your data in your database or drop entire table etc.)
Even you let them build a specific part of the query, it is still open to Sql Injection. If it is ok for all those dangers, then you may try the following.
This is and old script I used and let users set a specific part of the query. Basics are using string.Template and eval (the evil part)
Define your Model:
class SomeModel(Model):
usr = ForeingKey(User)
ct = ForeignKey(ContentType) # we will choose related DB table with this
extra_params = TextField() # store extra filtering criteria in here
Lets execute all queries belongs to a user. Say we have a User query with extra_params is_staff and 'username__iontains'
usr: somebody
ct: User
extra_params: is_staff=$stff_stat, username__icontains='$uname'
$ defines placeholders in extra_params
from string import Template
for _qry in SomeModel.objects.filter(usr='somebody'): # filter somebody's queries
cts = Template(_qry.extra_params) # take extras with Template
f_cts = cts.substitute(stff_stat=True, uname='Lennon') # sustitute placeholders with real time filtering values
# f_cts is now `is_staff=True, username__icontains='Lennon'`
qry = Template('_qry.ct.model_class().objects.filter($f_cts)') # Now, use Template again to place our extras into a django `filter` query. We also select related model in here with `_qry.ct.model_class()`
exec_qry = qry.substitute(f_cts=f_cts)
# now we have `User.objects.filter(is_staff=True, username__icontains='Lennon')
query = eval(exec_qry) # lets evaluate it!
If you have all relted imports done,then you an use Q or any other query building option in your extra_params. Also You can use other methods to form Create or Update queries.
You can read more about Template form there. But as I said. It is REALLY DANGEROUS to give a such option to other users.
Also you may need to read about Django Content Type
Update: As #GillBates mentioned, you can use a dictonary structure to create the query. In this case, you will not need Template anymore. You can use json for such data transfer (or any other if you wish). Assuming you use json to get the data from an outer source following code is a scratch that uses some variables from the upper code block.
input_data : '{"is_staff"=true, "username__icontains"="Lennon"}'
import json
_data = json.loads(input_data)
result_set = _qry.ct.model_class().objects.filter(**_data)

According to your answer,
User passes some content-specific parameters into a form, then view function, that recieves POST, constructs query
one option is to store parameters (pickle'd or json'ed, or in a model) and reconstruct query with regular django means. This is somewhat more robust solution, since it can handle some datastructure changes.

You could create a new model user_option and store the selections in this table.
From your question, it's hard to determine whether it is a better solution, but it would make your user's choices more explicit in your data structure.

pickle unicode strings with non-ascii caracters to mysql in django

Consider the I have an dictionary that I want to store in db using python's pickle.
My question is: which django models' field should I use?
So far I've been using a CharField, but there seems to be an error:
I pickle a u'\xe9' (i.e. 'É'), and I get:
Incorrect string value: '\xE1, ist...' for column 'edition' at row 1
(the ,"ist..." was because I have more text after the 'É').
I'm using
data = dict();
data['foo'] = input_that_has_the_caracter
to_save_in_db = cPickle.dumps(data)
Should I use a binary field and pickle with a protocol that uses binary? Because I have to change the db in order to do that, so it is better to be sure first...

You should check if you are using a proper encoding for your table AND column in your database backend (I'm assuming MySQL since your error message seems to be from it). In MySQL columns can have different encoding than the table. See if it's UTF-8.

Find which django model field contains a bad value

I'm new to stackoverflow and to python\django. I have already solved my problem, but I hoped that I can get help about how to solve it faster next time.
I have a very simple python function which copies table records from one db to another (sql server to sqllite). The table has hundreds of columns. When I save the model object to sqllite, django throws the following exception:
'utf8' codec can't decode byte ...
I understand that the data in one of the columns is problematic for utf8 conversion. What I wanted to know is what columns this is. I tried different approaches but eventually I had to write the following code to find the bad column:
build = Builds.objects.using('realdb').get(buildid=12524)
n = Builds()
for field in Builds._meta.fields:
val = getattr(build, field.name);
try:
setattr(n, field.name, val)
n.save(using="default")
except:
return HttpResponse(field.name + ": " + val.__str__())
It basically copies column values one be one to the new model object and stops when it encounters an error. Is there a better way to do this next time? I tried breaking on exception in PyCharm, but it breaks on all the many of exceptions thrown within django framework itself.
Alon.

I don't think there's any way of determining which specific field is causing the problem without resorting to testing each and every field as you're doing here. You can try to repair the problem fields instead of returning the error.
Take a look at this section of the unicode docs. Basically you can coerce the values by replacing the non-unicode portion or removing the non-unicode portion altogether.
Alternatively, if you know what encoding the strings are in, you can decode the string and re-encode appropriately using string.encode and string.decode respectively.