Intro
I have a cluster to monitor using Zabbix 2.0, everything works fine and I have all the data I need on Zabbix, but the way zabbix displays the data is not optimal for our use case. At the same time I have a python app running with a web front end I can use to create a more refined way of displaying Zabbix's data. What I want to do is to turn Zabbix's latest data tab into a grid view with a host in every row and the items as columns (like a spreadsheet).
The problem
Apparently Zabbix's API is still a work in progress and the interface sometimes changes, which should not be a problem if some basic functionality is working. What I need to do is to be able to fetch the list of hosts not only IDs but the host's info as well. And for each host I need to be able to fetch some items, again not only the items ID but the entire data too. So far I've tried using two Python libraries to do it: zabbix_api and PyZabbix, no luck so far since both libraries fetch only IDs and not the data I need for hosts and items.
The question
Is there a library/way of doing this that actually works or is this API in a too early stage yet?
Thanks in advance!
I use zabbix_api to do navigate through zabbix catalogs, get hosts, get host, get host's items, etc. Though I didn't try to get the data with python, I don't see why it shouldn't work. I do get data from PHP using PhpZabbixApi. Any specific problems you've run into?
PyZabbix is vital and pretty usable. In fact it is 1:1 mapping of Zabbix API to Python.
Related
I am trying to convert my BASH script to Python and am having difficulties in finding the equivalent code for openstack server show or openstack server list --long. I would like to know what host is my server currently located and use this information for a check before migrating it to another host.
Looking through the latest novaclient documentation and its servers module, I have found two potential commands that I was hoping would accomplish the task, but does not do so:
list(detailed=True)
Gets a list servers
detailed=True should return detailed server info (optional).
This returns a regular list of servers with their names.
get(server)
Get a server
This returns only the name of the server.
I have been researching for the past two days, and I could not find the same / similar problem here in stack overflow so I have decided to ask and I am hoping that someone can help me with this.
Either list or get should be fine here.
As an example get would be used like this.
instance = nova_client.servers.get('my-server')
print(instance.name)
print(instance.addresses)
print(instance.status)
Or using list.
for instance in nova_client.servers.list():
print(instance.name)
print(instance.addresses)
print(instance.status)
If you want an easy way of understanding the type of data you can get, you can simply use the Python inbuilt dir.
instance = nova_client.servers.get('my-server')
print(dir(instance))
'my-server' needs to be the id as in instance.id, the name of the server is not valid.
I cant yet comment, so i wrote an answer.
The closest I've gotten is via the rest API call.
https://{HOST}/rest/api/2/project/{Project Key}/statuses
But I need the same call via Python. But I'm unable to find an adequate way.
The closest I've gotten in Python is
jiraInstance.statuses() but this returns all possible statuses for our Jira site.
I need to narrow it down to the workflows for a specific project.
Any help would be appreciated.
Background
This is for a reporting tool where we create a table with all the defects for the specific project in question. In python I can currently retrieve all the statuses/priorities for the Bugs/Bug-task issues but that only returns statuses for the existing bugs. I require a way to retrieve all the statuses from a workflow of the specific project.
This will list all project keys in a given project with their respective status...
issues = jira.search_issues('project=projectname')
for issue in issues:
print (issue.key, 'Status: ',issue.fields.status)
I have a basic personal project website that I am looking to learn some web dev fundamentals with and database (SQL) fundamentals as well (If SQL is even the right technology to use??).
I have the basic skeleton up and running but as I am new to this, I want to make sure I am doing it in the most efficient and "correct" way possible.
Currently the site has a main index (landing) page and from there the user can select one of a few subpages. For the sake of understanding, each of these sub pages represents a different surf break and they each display relevant info about that particular break i.e. wave height, wind, tide.
As I have already been able to successfully scrape this data, my main questions revolve around how would I go about inserting this data into a database for future use (historical graphs, trends)? How would I ensure data is added to this database in a continuous manner (once/day)? How would I use data that was scraped from an earlier time, say at noon, to be displayed/used at 12:05 PM rather than scraping it again?
Any other tips, guidance, or resources you can point me to are much appreciated.
This kind of data is called time series. There are specialized database engines for time series, but with a not-extreme volume of observations - (timestamp, wave heigh, wind, tide, which break it is) tuples - a SQL database will be perfectly fine.
Try to model your data as a table in Postgres or MySQL. Start by making a table and manually inserting some fake data in a GUI client for your database. When it looks right, you have your schema. The corresponding CREATE TABLE statement is your DDL. You should be able to write SELECT queries against your table that yield the data you want to show on your webapp. If these queries are awkward, it's a sign that your schema needs revision. Save your DDL. It's (sort of) part of your source code. I imagine two tables: a listing of surf breaks, and a listing of observations. Each row in the listing of observations would reference the listing of surf breaks. If you're on a Mac, Sequel Pro is a decent tool for playing around with a MySQL database, and playing around is probably the best way to learn to use one.
Next, try to insert data to the table from a Python script. Starting with fake data is fine, but mold your Python script to read from your upstream source (the result of scraping) and insert into the table. What does your scraping code output? Is it a function you can call? A CSV you can read? That'll dictate how this script works.
It'll help if this import script is idempotent: you can run it multiple times and it won't make a mess by inserting duplicate rows. It'll also help if this is incremental: once your dataset grows large, it will be very expensive to recompute the whole thing. Try to deal with importing a specific interval at a time. A command-line tool is fine. You can specify the interval as a command-line argument, or figure out out from the current time.
The general problem here, loading data from one system into another on a regular schedule, is called ETL. You have a very simple case of it, and can use very simple tools, but if you want to read about it, that's what it's called. If instead you could get a continuous stream of observations - say, straight from the sensors - you would have a streaming ingestion problem.
You can use the Linux subsystem cron to make this script run on a schedule. You'll want to know whether it ran successfully - this opens a whole other can of worms about monitoring and alerting. There are various open-source systems that will let you emit metrics from your programs, basically a "hey, this happened" tick, see these metrics plotted on graphs, and ask to be emailed/texted/paged if something is happening too frequently or too infrequently. (These systems are, incidentally, one of the main applications of time-series databases). Don't get bogged down with this upfront, but keep it in mind. Statsd, Grafana, and Prometheus are some names to get you started Googling in this direction. You could also simply have your script send an email on success or failure, but people tend to start ignoring such emails.
You'll have written some functions to interact with your database engine. Extract these in a Python module. This forms the basis of your Data Access Layer. Reuse it in your Flask application. This will be easiest if you keep all this stuff in the same Git repository. You can use your chosen database engine's Python client directly, or you can use an abstraction layer like SQLAlchemy. This decision is controversial and people will have opinions, but just pick one. Whatever database API you pick, please learn what a SQL injection attack is and how to use user-supplied data in queries without opening yourself up to SQL injection. Your database API's documentation should cover the latter.
The / page of your Flask application will be based on a SQL query like SELECT * FROM surf_breaks. Render a link to the break-specific page for each one.
You'll have another page like /breaks/n where n identifies a surf break (an integer that increments as you insert surf break rows is customary). This page will be based on a query like SELECT * FROM observations WHERE surf_break_id = n. In each case, you'll call functions in your Data Access Layer for a list of rows, and then in a template, iterate through those rows and render some HTML. There are various Javascript and Python graphing libraries you can feed this list of rows into and get graphs out of (client side or server side). If you're interested in something like a week-over-week change, you should be able to express that in one SQL query and get that dataset directly from the database engine.
For performance, try not to get in a situation where more than one SQL query happens during a page load. By default, you'll be doing some unnecessary work by going back to the database and recomputing the page every time someone requests it. If this becomes a problem, you can add a reverse proxy cache in front of your Flask app. In your case this is easy, since nothing users do to the app cause its content to change. Simply invalidate the cache when you import new data.
We are building an exensive api-link with the Exact online odata API. Problem we are having is that many objects cant be updated or deleted. For instance BankEntryLines, GeneralJournalEntryLines.
We have now worked around this by creating new EntryLines upon each update or delete, but this creates much unclarity in some cases.
Can the API be changed, or can I get extra authorization to be able to update or delete these objects, just like is possible in the GUI?
As the Exact Online REST API doesn't support modifying on quite some objects, there is no way to achieve what you want using the REST API. If the Exact Online XML API doesn't support updating either, there is only one solution left.
That solution is forbidden by Exact, and it could risk you lose you application developer status. You can make those changes using HTTP POSTS on the web site itself. If you can extract the calls that are made through the screens, you can mimic their behavior and by replaying that, you can modify what you need.
If you want to make a coupling to Exact Online and you are starting with developing, I want to suggest you to take a look at Invantive Data Hub, which allows updating Exact Online using SQL syntax. (To give full disclosure: I work for that company)
At my work, we use Oracle for our database. Which works great. I am not the main db admin, but I do work with it. One thing I like is that the DB has a built in logic layer using PL/SQL which ca handle logic related to saving the data and retrieve it. I really like this because it allows our MVC application (PHP/Zend Framework) to be lighter, and makes it easier to tie in another platform into the data, such as desktop or mobile.
Although, I have a personal project where I want to use couchdb or mongodb, and I want to try and accomplish a similar goal. outside of the mvc/framework, I want to have an API layer that the main applications talk to. they dont actually talk directly to the database. They specify the design document (couchdb) or something similar for mongo, to get the results. And that API layer will validate the incoming data and make sure that data itself is saved and updated properly. Such as saving a new user, in the framework I only need to send a json obejct with the keys/values that need to be saved and the api layer saves the data in the proper places where needed.
This API would probably have a UI, but only for administrative purposes and to make my life easier. In general it will always reply with json strings, or pre-rendered/cached html in some cases. Since each api layer would be specific to the application anyways.
I was wondering if anyone has done anything like this, or had any tips on nethods I could accomplish this. I am currently looking to write my application in python, and the front end will likely be something like Angularjs. Although I am also looking at node.js for a back end.
We do this exact thing at my current job. We have MongoDB on the back end, a RESTful API on top of it and then PHP/Zend on the front end.
Most of our data is read only, so we import that data into MongoDB and then the RESTful API (in Java) just serves it up.
Some things to think about with this approach:
Write generic sorting/paging logic in your API. You'll need this for lists of data. The user can pass in things like http://yourapi.com/entity/1?pageSize=10&page=3.
Make sure to create appropriate indexes in Mongo to match what people will query on. Imagine you are storing users. Make an index in Mongo on the user id field, or just use the _id field that is already indexed in all your calls.
Make sure to include all relevant data in a given document. Mongo doesn't do joins like you're used to in Oracle. Just keep in mind modeling data is very different with a document database.
You seem to want to write a layer (the middle tier API) that is database agnostic. That's a good goal. Just be careful not to let Mongo specific terminology creep into your exposed API. Mongo has specific operators/concepts that you'll need to mask with more generic terms. For example, they have a $set operator. Don't expose that directly.
Finally after having a decent amount of experience with CouchDB and Mongo, I'd definitely go with Mongo.