Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
For a class, I would like to automatically evaluate (parts) of the coding assignments of students. The setup I had in mind is something like:
Students get a class skeleton, which they have to fill in.
Students ``upload'' this class definition to a server (or via webinterface)
The server runs a script an test on specific functions, eg class.sigmoid(x), and checks if the output of the function is correct and might give suggestions.
This setup brings a whole lot of problems, since you're evaluating untrusted code. However, it would be extremely useful, for many of my classes, so I'm willing to spend some time in thinking it trough. I remember Coursera had something similar for matlab/octace assignments, but I can't get the details of that.
I've looked at many online python interfaces (eg, codecademy.com, ideone.com, c9.io); while they seem perfect to learn and or share code, with online evaluation. I do miss the option, that the evaluation script is "hidden" from the students (ie the evaluation script should contain a correct reference implementation to compare output on random generated data). Moreover, the course I give requires some data (eg images) and packages (sklearn / numpy), which is not always available.
Specifically, my questions are
Do I miss an online environment which actually offers such a functionality. (that would be easiest)
To set this up myself, I was thinking to host it at (eg) amazon cloud (so no problem with infrastructure at University), but are there any python practices you could recommend on sandboxing the evaluation?
Thanks in advance for any suggestions!
Pity to hear that the question is not suitable for StackOverflow. Thanks to the people (partially) answering the question.
After some more feedback via other channels, I think my approach will become as follows:
Student gets skeleton and fills it in
Student also has the evaluation script.
In the script, some connections with a server are made to
login
obtain some random data
check if the output of the students code is numerically identical to what the server expects.
In this way the students code is evaluated locally, but only output is send to the server. This limits the kind of evaluations possible, but still allows for kind of automatic evaluation of code.
Sandboxing Python in general is impossible. You can try to prevent dangerous operations, which will mean significantly limiting what the student code can do. But that will likely leave open attack vectors anyway. A better option is to use OS-level sandboxing to isolate the Python process. The CodeJail library uses AppArmor to provide a safe Python eval, for example.
As an example of the difficulty of sandboxing Python, see Eval really is dangerous, or consider this input to your sandbox: 9**9**99, which will attempt to compute an integer on the order of a googolplex, consuming all of your RAM after a long time.
This is currently a very active field in programming languages research.
I know of these two different approaches that look at the problem:
- http://arxiv.org/pdf/1409.0166.pdf
- http://research.microsoft.com/en-us/um/people/sumitg/pubs/cacm14.pdf (this is actually only one of very many papers by Sumit and his group)
You may want to look at these things to find something that could help with your problem (and edit this answer to make it more useful).
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I have observed that staticians and machine learning scientist generally doesnt follow OOPS for ML/data science projects when using Python (or other languages).
Mostly it should be due to lack of understanding of best software engineering practises in oops while developing ML code for production. Because they mostly come from math & statistics education background than computer science.
Days when ML scientist develop ad hoc protype code and another software team make it production ready are over in the industry.
Questions
How do we structure code using OOP for ML project?
Should every major task (from picture above) like data cleaning, feature transformation, grid search, model validation etc. be a individual class? What are the recommended code design practises for ML?
Any good github links with well strcutured code for reference (may be a well written kaggle solution)
Should every class like data cleaning have fit(), transform(), fit_transform() function for every process like remove_missing(), outlier_removal()? When this is done why is scikit-learn BaseEstimator be usually inherited?
What should be the structure of typical config file for ML projects in production?
You are right about one thing being special about ML: data scientists are generally clever people, so they have no problem in presenting their ideas in code. The problem is that they tend to create fire&forget code, because they lack software development craftsmanship - but ideally this shouldn't be the case.
When writing code it shouldn't make any difference what the code is for1. ML is just another domain like anything else, and should follow clean code principles.
The most important aspect always should be SOLID. Many important aspects directly follow: maintainability, readability, flexibility, testability, reliability etc. What you can add to this mix of features is risk of change. It doesn't matter whether a piece of code is pure ML, or banking business logic, or audiological algorithm for a hearing instrument. All the same - the implementation will be read by other developers, will contain bugs to fix, will be tested (hopefully) and possibly refactored and extended.
Let me try to explain this in more detail while addressing some of your questions:
1,2) You shouldn't think that OOP is the goal in itself. If there is a concept that can be modeled as a class and this will make its usage easy for other developers, it will be readable, easy to extend, easy to test, easy to avoid bugs then of course - make it a class. But unless it's needed, you shouldn't follow the BDUF antipattern. Start with free functions and evolve into a better interface if needed.
4) Such complex inheritance hierarchies are typically created to allow implementation to be extensible (see "O" from SOLID). In this case, library users can inherit BaseEstimator and it's easy to see what methods can they override and how this will fit into scikit's existing structure.
5) Almost the same principles as for code, but with people who will create/edit these config files in mind. What will be the easiest format for them? How to choose parameter names so it will be obvious what do they mean, even for a beginner, who is just starting to use your product?
All these things should be combined with guessing how likely is it that this piece of code will be changed/extended in the future? If you are sure something should be written in stone, don't worry about all aspects too much (e.g. focus only on readability), and direct your efforts to more critical parts of the implementation.
To sum up: think about people who will interact with what you create in the future. In case of products/config files/user interfaces it should be always "user first". In case of code, try to put yourself in the shoes of a future developer who will want to fix/extend/understand your code.
1 There are of course some special cases, like code that needs to be formally proven correct or extensively documented because of formal regulations and this main goal imposes some particular constructs/practices.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I've always have trouble with dynamic language like Python.
Several troubles:
Typo error, I can use pylint to reduce some of these errors. But there's still some errors that pylint can not figure out.
Object type error, I often forgot what type of the parameter is, int? str? some object? Also, forgot the type of some object in my code.
Unit test might help me sometimes, but I'm not always have enough time to do UT. When I need a script to do a small job, the line of code are 100 - 200 lines, not big, but I don't have time to do the unit test, because I need to use the script as soon as possible. So, many errors appear.
So, any idea on how to reduce the number of these troubles?
Unit testing is the best way to handle this. If you think the testing is taking too much time, ask yourself how much time you are loosing on defects - identifying, diagnosing and rectifying - after you have released the code.
In effect, you are testing in production, and there's plenty of evidence to show that defects found later in the development cycle can be orders of magnitude more expensive to fix.
In addition to unittesting (see chamila_c's answer), sticking to good conventions and coding style helps. I think I know the sort of one-use scripts you're talking about (assuming that is what you're talking about), and often writing a full test suite for them seems like overkill. A few other tips which might help:
Break your code up into functions. It is often easier to identify a problem, especially a naming problem if you are dealing with a small, isolated piece of code. It is also much easier to write a unit test for small function. I find that using this approach means you don't need a full testing suite to test and isolate an identified problem.
Stick to a consistent, and expressive naming convention. For example, use min_value = min(all_values) rather than something arbitrary like a = min(b). Same goes for function names, use def calculate_mean(sequence) rather than def f(s)
Read and apply PEP8.
Document your code with comments. This only takes a few seconds while writing the code, and it makes it much easier to figure out what's going on when you come back to it. Its amazing the detail you can forget just in one day!
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
First I want to clearify that I mean by reverse engineering something like "decompiling" and getting back the original source code or something similiar.
Yesterday I read a question about someone who wanted to protect his python code from "getting stolen" in other words: he didn't like that someone can read his python code.
The interesting thing I read was that someone said that the only reliable way to "protect" his code from getting reverse engineered is by using a Webservice.
So I could actually only write some GUIs in Python, PHP, whatever and do the "very secret code" I want to protect via a Webservice. (Basically sending variables to the host and getting results back).
Is it really impossible to reverse engineer a Webservice (via code and without hacking into the Server)? Will this be the future of modern commercial applications? The cloud-hype is already here. So I wouldn't wonder.
I'm very sorry if this topic was already discussed, but I couldn't find any resources about this.
EDIT: The whole idea reminds me of AJAX. The code is executed on the server and the content is sent to the client and "prettified". The client himself doesnt see what php-code or other technology is behind.
Wow, this is awesome! I've never thought it this way, but you could create a program that crawls an api, and returns as an output a django/tastypie software that mimics everything the api does.
By calling the service, and reading what it says, you can parse it, and begin to see the relationships between objects inside the api. Having this, you can create the models, and tastypie takes it from this point.
The awesome thing about this, is that normal people (or at least not backend developers) could create an api just by describing what they want to be as an output. I've seen many android/iphone developers creating a bunch of static xml or json, so they can call their service, and start the frontend development. Well what if that was enough? Take some xml/json files as input, get a backend as an output.
Yes,
All they could do is treat your web service as a black box, query the WSDL for all the parameters it accepts and the data that it returns.
They could then submit different variables and see what different results are. The "code" could not be seen or stolen (with proper security) but the inputs and outputs could be duplicated.
If you want to secure your "very secret code" a web service is a great way to protect the actual code.
-sb
It depends on what you mean by reverse engineering: by repeatedly sending input and analyzing the output the behaviour of your code can still be seen. I wouldn't have your code but I can still see what the system does. This means I could build a similar system that does the same thing, given the same input.
It would be hard to catch exceptional cases (such as output that is different on one day of the year only) but the common behaviour can certainly be copied. It is similar to analyzing the protocol of an instant messaging client: you may not have the original code but you can still build a copy.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I've written a few little things in python, and I am ramping up to build something a little more challenging.
The last project I made basically ingested some text files, did some regex over each file and structured the data in an useful way so I could investigate some data I have.
I found it quite tough near the end to remember what section operated on what part of the text, especially as the code grew as I 'fixed' things along the way.
In my head, I imagine my code to be a series of small interconnected modules - descrete .py files that I can leave to one side knowing what they do, and how they interoperate.
The colleague that showed me how to def functions basically meant that I ended up with one really long piece of code that I found really hard to navigate and troubleshoot.
(1) Is this the right way? or is there an easier way of making modules that pass variables between them, I think i would find this better, as I could visualise the flow better (mainly becuase its how I was used to working in MATLAB a few years ago I guess)
(2) Can you use this method to plan out the various layers of functions before hand to give you a 'map' to write towards?
(3) is there any easy to access tutorials for this kind of stuff? I often find the tutorials suddenly jump way over my head....
Thanks.
(1) It is possible to write a fine programme in a single .py file
(2) In any style of programming, it is always (apart from special, hardware-driven cases) best to break your code up into short functions (or methods) that accomplish a discrete task.
(3) Experienced programmers will frequent write their code one way, discover a problem, either write more code, or different code, and consider whether any of their existing code can be broken out into a separate function.
A sign that you need to do this is when you are sequentially assigning to variables to pass data down your function. Never copy-paste your code to another place, even with changes, unless it be to break it out as a function, and replace the original code with a call to that function.
(4) In many cases, it can be useful to organise your code into classes and objects, even when it is not technologically necessary to do so. It can help you see that you have defined a complete set of operations (or not) necessary on some collection of data.
(5) Programming is actually quite hard. Even among those who have a talent for it, it takes a while to be comfortable. As an illustration, when I was doing my master's degree, I and my (fairly talented) friends all felt only in our final year that we had begun to achieve a degree of facility and competence (and these are all people who had been programming since at least their teenage years).
The important thing is to keep learning and improving, rather than repeating the same one or two years of experience over and over.
(6) To that end, read books and articles. Try new things. Think.
Others have suggested studying other experienced programmers' code from open source projects, etc. and from tutorials and textbooks, which is sound advice. Sometimes a similar example is all you need to set you on the right path.
I also suggest to use your own frustration and experience as feedback to help yourself improve. Whenever you find yourself thinking any of the following:
It feels like I'm writing the same code over and over again with only small changes
I wrote this code myself, but I had to study it for a long time to re-learn how it works
Each time I go back and add something to this code it takes me longer to get it working again
There's a bug in this code somewhere, but I haven't a clue where
Surely somebody somewhere has solved this problem already
Why is this taking me so long to get done?
That means you have room for improvement in your technique. A lot of the difference between an expert and beginning programmer is the ability to do the following:
Don't Repeat Yourself (DRY): Instead of copy-pasting code, or writing the same code over and over with variations, write a single common routine with one or more parameters that can do all of those things. Then call that routine in multiple places.
Keep It Simple (KIS): Break up your code into simple well-defined behaviors/routines that make sense on their own, organized into classes/modules/packages, so that each part of the overall program is easy to understand and maintain. Write informative and concise comments, and document the calls even if you don't intend to publish them.
Divide & Conquer Testing: Thoroughly test each individual class, function, etc. by itself (preferably with a unit-testing framework) as you develop it, rather than only testing the entire application.
Don't Re-invent the Wheel: Use open source frameworks or other tools where possible to solve problems that are general and not specific to your application. In all but the most trivial cases, there is a risk that you do not fully understand the problem and your home-grown solution may be lacking in an important way.
Estimate Honestly: Study your own previous efforts to learn how long it takes you to do certain things. Try to work faster next time, but don't assume you will. Measure yourself and use your own experience to estimate future effort. Set expectations and bargain with scope.
It's hard to know even where to begin answering your question without a snippet of your code for reference. You might want to post your code to a free public site such as http://www.bitbucket.org/ or http://www.github.org/ and then include some specific questions about small snippets of code with links back to your repository. This allows respondents here to look at the code and comment on it. (Both of these options even include color syntax highlighting, and interested correspondent can even pull the code down, make changes and push up a patch or create their own branch of your code and send you a "pull" request so you can look at the differences and pull selected changesets back into your branch).
More generally there are a number of approaches to program design. You seem to be trying to re-invent a very old methodology which is referred to as "functional decomposition" --- look at the overall task at hand as a function (digest text files) and consider how that breaks down (decomposes) into smaller functions (ingest input files, parse them, prepare results, output those) and then breaking those down further until you have units which are small enough to be coded easily in your programming environment (Python).
Modern approaches (and tools) tend to use object oriented design methodologies. You might try reading: http://www.itmaybeahack.com/homepage/books/oodesign/build-python/html/index.html
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I teach undergraduate statistics, and am interested in administering personalized online assignments. I have already solved one portion of the puzzle, the generation of multiple version of a question using latex/markdown + knitr/sweave, using seeds.
I am now interested in developing a web-based system, that would use the various versions generated, and administer a different one for each student, online. I have looked into several sites related to forms (google docs, wufoo, formsite etc.), but none of them allow programmatic creation of questionnaires.
I am tagging this with R since that is the language I am most familiar with, and is key to solving the first part of the problem. I know that there are several web-based frameworks for R, and was wondering whether any of them are suitable for this job.
I am not averse to solutions in other languages like Ruby, Python etc. But the key consideration is the ability to programatically deliver online assignments. I am aware of tools like WebWork, but they require the use of Perl and the interfaces are usually quite clunky.
Feel free to add tags to the post, if you think I have missed a framework that would be more suitable.
EDIT. Let me make it clear by giving an example. Currently, if I want to administer an assignment online, I could simply create a Google Form, send the link to my students, and collect all responses in a spreadsheet, and automatically grade it. This works, if I just have one version of the assignment.
My questions is, if I want to administer a different version of the assignment for each student, and collect their responses, how can I do that?
The way you have worded your question it's not really clear why you have to mark the students' work online. Especially since you say that you generate assignments using sweave. If you use R to generate the (randomised) questions, then you really have to use R to mark them (or output the data set).
For my courses, I use a couple of strategies.
For the end of year exam (~500 students), each student gets a unique data set. The students log on to a simple web-site (we use blackboard since the University already has it set up). All students answer the same questions, but use their own unique data set. For example, "What is the mean". The answers are marked offline using an R script.
In my introductory R course, students upload their R functions and I run and mark them off line. I use sweave to generate a unique pdf for each student. Their pdf shows where they lost marks. For example, they didn't use the correct named arguments.
Coupling a simple web-form with marking offline gives you a lot of flexibility and is fairly straightforward.
I found one possible solution that might work using the RGoogleDocs package. I am posting this as an answer only because it is long. I am still interested in better approaches, and hence will keep the question open.
Here is the gist of the idea, which is still untested.
Create multiple versions of each assignment using knitr/Sweave.
Upload them to GoogleDocs using uploadDoc.
Share one document per student using setAccess which modifies access controls.
Create a common Google Form to capture final answers for each student.
The advantage I see is two-fold. One, since all final answers get captured on a spreadsheet, I can access them with R and grade them automatically. Two, since I have access to all the completed assignments on Google Docs, I can skim through them and provide individual comments as required (or let some of my TAs do it).
I will provide an update, if I manage to get this working, and maybe even create an R package if it would be useful for others.
I know that this was asked a long time ago, but I think that today the best solution is the package exams plus Moodle.
The package exams can now generate XML Moodle questions that can be upload to Moodle platform as the students can solve the exercices on-line.
This is an example of a question made with exams package and uploaded to Moodle.
i just stumbled upon the ?exams package in R: Link to the CRAN site. could this be something for you?