i have an app ( google app engine + high replication datastore ) which was not using eventual consistency ( high replication ) up till now and all my test worked perfectly.
now, for local testing in high replication, as soon as i moved to eventual consistency, they begin to fail. how do i prevent that ? or how do i test that part ?
i need it for x-entity transaction.
i am using something similar to https://developers.google.com/appengine/docs/python/tools/localunittesting#Writing_HRD_Datastore_Tests
edit:
I need to test the code correctly. The problem I have is with the testing part. How Anyone test eventual consistency ?
edit 1:
I have temporarily solved the problem with using probability=100% in above linked example. But Ideas are welcome.
Fix the failures.
Since you have no code and are very vague, it's hard to answer your question. But essentially either your app code or your tests are not taking into account the eventual consistency (ie, a query may not return with a value that was just updated in the database). When you turned on eventual consistency in the datastore, the query results you get will be different.
You either need to update your code to handle the eventual consistency situations with transactions, or update your tests to expect eventual consistency results.
edit
This question is still too general. It depends if you're doing, say functional or system testing. Are you looking for particular results? Or just an HTTP status=200?
In general, like all testing, you need to identify what constitutes success and what constitutes a failure case. In a given situation, is it acceptable for old data to appear? In that case, the test should succeed with either the old or new values.
I'd recommend starting out considering whether you want to run deterministic or non-deterministic tests. For deterministic tests, you'd essentially want to run through the same tests with probability=0 and probability=100, and ensure you get the correct values for both.
I haven't figured out how to write non-deterministic tests in a completely useful manner, other than as a stress test. You can verify that certain required values are met, and other eventually-consistent values fall within a valid range. This is a lot of work, because most likely you have a range of values that may depend on another range of values, and since your final output may consist of both, you'll have to validate that the combinations are correct - essentially you end up reproducing some of your application logic if you really want to verify everything is correct.
The situation you are facing is one of the drawbacks (or call it a feature) of High Replication Data Store. Usually these situations are tackled via transparent caching using memcache. If you had prior experience working with a db master/slave architecture, slave lags are tackled in a similar manner.
Related
Can anyone tell the difference between pyspark.sql.functions.approxCountDistinct (I know it is deprecated) and pyspark.sql.functions.approx_count_distinct? I have used both versions in a project and have experienced different values
As you mentioned it, pyspark.sql.functions.approxCountDistinct is deprecated. The reason is most likely just a style concern. They probably wanted everything to be in snake case. As you can see in the source code pyspark.sql.functions.approxCountDistinct simply calls pyspark.sql.functions.approx_count_distinct, nothing more except giving you a warning. So regardless the one you use, the very same code runs in the end.
Also, still according to the source code, approx_count_distinct is based on the HyperLogLog++ algorithm. I am not very familiar with the algorithm but it is based on repetitive set merging. Therefore, the result will most likely depend on the order in which the various results of the executors are merged. Since this is not deterministic with spark, this could explain why you witness different results.
I just migrated a pytest test suite from quickcheck to hypothesis. This worked quite well (and immediately uncovered some hidden edge case bugs), but one major difference I see is related to test isolation between the two property managers.
quickcheck seems to simply run the test function multiple times with different parameter values, each time running my function-scoped fixtures. This also results in many more dots in pytest's output.
hypothesis however seems to run only the body of the test function multiple times, meaning for example no transaction rollbacks between individual runs. This then means I cannot reliably assert for a number of DB entries when my test inserts something into the DB for example, since all the entries from the previous run would still be hanging around.
Am I missing something obvious here or is this expected behaviour? If so, is there a way to get the number of runs hypothesis has done as a variable to use inside the test?
I'm afraid you're a bit stuck and there isn't currently any good solution to this problem.
The way Hypothesis needs to work (which is the source of a lot of its improvements over pytest-quickcheck) doesn't meet pytest's assumptions about test execution. The problem is mostly on the pytest side - the current pytest fixture system has some very baked in assumptions about how you run a test that do not play well with taking control of the test execution, and the last time I tried to work around this I ended up sinking about a week of work into it before giving up and basically saying that either something needs to change on the pytest side or someone needs to fund this work if it's going to get any better.
I'm trying to achieve strong consistency with ndb using python.
And looks like I'm missing something as my reads behave like they're not strongly consistent.
The query is:
links = Link.query(ancestor=lead_key).filter(Link.last_status ==
None).fetch(keys_only=True)
if links:
do_action()
The key structure is:
Lead root (generic key) -> Lead -> Website (one per lead) -> Link
I have many tasks that are executed concurrently using TaskQueue and this query is performed at the end of every task. Sometimes I'm getting "too much contention" exception when updating the last_status field but I deal with it using retries. Can it break strong consistency?
The expected behavior is having do_action() called when there are no links left with last_status equal to None. The actual behavior is inconsistent: sometimes do_action() is called twice and sometimes not called at all.
Using an ancestor key to get strong consistency has a limitation: you're limited to one update per second per entity group. One way to work around this is to shard the entity groups. Sharding Counters describes the technique. It's an old article, but as far as I know, the advise is still sound.
Adding to Dave's answer which is the 1st thing to check.
One thing which isn't well documented and can be a bit surprising is that the contention can be caused by read operations as well, not only by the write ones.
Whenever a transaction starts the entity groups being accessed (by read or write ops, doesn't matter) are marked as such. The too much contention error indicates that too many parallel transactions simultaneously try to access the same entity group. It can happen even if none of the transactions actually attempts to write!
Note: this contention is NOT emulated by the development server, it can only be seen when deployed on GAE, with the real datastore!
What can add to the confusion is the automatic re-tries of the transactions, which can happen after both actual write conflicts or just plain access contention. These retries may appear to the end-user as suspicious repeated execution of some code paths - which I suspect could explain your reports of do_action() being called twice.
Usually when you run into such problems you have to re-visit your data structures and/or the way you're accessing them (your transactions). In addition to solutions maintaining the strong consistency (which can be quite expensive) you may want to re-check if consistency is actually a must. In some cases it's added as a blanket requirement just because appears to simplify things. From my experience it doesn't :)
There is nothing in your sample that ensures that your code is only called once.
For the moment, I am going to assume that your "do_action" function does something to the Link entities, specifically that it sets the "last_status" property.
If you do not perform the query and the write to the Link Entity inside a transaction, then it is possible for two different requests (task queue tasks) to get results back from the query, then both write their new value to the Link entity (with the last write overwriting the previous value).
Remember that even if you do use a transaction, you don't know until the transaction is successfully completed that nobody else tried to perform a write. This is important if you are trying to do something external to datastore (for example, making a http request to an external system), as you may see http requests from transactions that would eventually fail with a concurrent modification exception.
I'm writing unit tests that have a database dependency (so technically they're functional tests). Often these tests not only rely on the database to be live and functional, but they can also rely on certain data to be available.
For example, in one test I might query the database to retrieve sample data that I am going to use to test the update or delete functionality. If data doesn't already exist, then this isn't exactly a failure in this context. I'm only concerned about the pass/fail status of the update or delete, and in this situation we didn't even get far enough to test it. So I don't want to give a false positive or false negative.
Is there an elegant way to have the unit test return a 3rd possible result? Such as a warning?
In general I think the advice by Paul Becotte is best for most cases:
This is a failure though- your tests failed to set up the system in
the way that your test required. Saying "the data wasn't there, so it
is okay for this test to fail" is saying that you don't really care
about whether the functionality you are testing works. Make your test
reliably insert the data immediately before retrieving it, which will
give you better insight into the state of your system.
However, in my particular case, I am writing a functional test that relies on data generated and manipulated from several processes. Generating it quickly at the beginning of the test just isn't practical (at least yet).
What I ultimately found to work as I need it to use skipTest as mentioned here:
Skip unittest if some-condition in SetUpClass fails
I've read quite a few answers on here about testing helper methods (not necessarily private) in my unit tests and I'm still not quite sure what the best approach should be for my current situation.
I currently have a block of logic that runs as a scheduled job. It does a number of mostly related things like update local repositories, convert file types, commit these to other repos, clean up old repos, etc. I need all of this code to run in a specific order, so rather than setting a bunch of scheduled jobs, I took a lot of these small methods and put them into one large method that would enforce the order in which the code is run:
def mainJob():
sync_repos()
convert_files()
commit_changes()
and so on. Now I'm not sure how to write my tests for this thing. It's frustrating to test the entire mainJob() function because it does so many things and is really more of a reliability feature anyway. I see a lot of people saying I should only test the public interface, but I worry that there will potentially be code that isn't directly verified.