Slashdot Mirror

← Back to Stories (view on slashdot.org)

Why Jupyter is Data Scientists' Computational Notebook of Choice (nature.com)

Posted by msmash on from the growing-a-loyal-base dept.

Jeffrey M. Perkel, writing for Nature: Perched atop the Cerro Pachon ridge in the Chilean Andes is a building site that will eventually become the Large Synoptic Survey Telescope (LSST). When it comes online in 2022, the telescope will generate terabytes of data each night as it surveys the southern skies automatically. And to crunch those data, astronomers will use a familiar and increasingly popular tool: the Jupyter notebook. Jupyter is a free, open-source, interactive web tool known as a computational notebook, which researchers can use to combine software code, computational output, explanatory text and multimedia resources in a single document. Computational notebooks have been around for decades, but Jupyter in particular has exploded in popularity over the past couple of years. This rapid uptake has been aided by an enthusiastic community of user-developers and a redesigned architecture that allows the notebook to speak dozens of programming languages -- a fact reflected in its name, which was inspired, according to co-founder Fernando Perez, by the programming languages Julia (Ju), Python (Py) and R.

[...] For data scientists, Jupyter has emerged as a de facto standard, says Lorena Barba, a mechanical and aeronautical engineer at George Washington University in Washington DC. Mario Juric, an astronomer at the University of Washington in Seattle who coordinates the LSST's data-management team, says: "I've never seen any migration this fast. It's just amazing." Computational notebooks are essentially laboratory notebooks for scientific computing. Instead of pasting, say, DNA gels alongside lab protocols, researchers embed code, data and text to document their computational methods. The result, says Jupyter co-creator Brian Granger at California Polytechnic State University in San Luis Obispo, is a "computational narrative" -- a document that allows researchers to supplement their code and data with analysis, hypotheses and conjecture. For data scientists, that format can drive exploration.

15 of 58 comments (clear)

  1. Wasn't this recently discussed? by Anonymous Coward · · Score: 5, Interesting

    Economics Nobel Laureate Paul Romer Is a Python Programming Convert

  2. Huge Notebook fan. by 0100010001010011 · · Score: 4, Interesting

    Anecdotal, but I do 90% of my python 'development' in Jupyter Notebooks.

    For work I can make a nice notebook and have it generate a PDF for archiving. It'll output to LaTeX, html, .py and a number of other formats.

    Now you can include multiple languages in the same notebook including R and Matlab, both popular in their own niches of use.

    1. Re:Huge Notebook fan. by Lab+Rat+Jason · · Score: 5, Interesting

      For the past year, I've begun using Jupyter and although I like it, there are some features that really bother me, and worry me when it comes time to create reproducible science. 1) Jupyter doesn't integrate automatically with any kind of source control software, and in the circles I run in, it is largely ignored. Data scientists act like they've never heard of source control, and what makes it worse, my local university is pumping out student after student where they introduce them to data science with Jupyter, but never bring up the topic of coding standards and recoverability. 2) Jupyter allows you to execute cells out of order. While this definitely helps speed up development (when you make a mistake, and just want to fix the relevant line and continue, rather than re-loading your entire data set), it presents a unique risk when someone thinks they've discovered something amazing, only to be unable to reproduce it after a restart, or when sharing the notebook with someone else. This can happen when race conditions exist, or when code makes changes to the database, and your out of order execution causes spooky behavior. 3) Jupyter doesn't encourage enterprise deployment. Too often I see experimental data science done well, but due to the nature of rapid development, nothing is modular, nothing is object oriented, and so if the solution was a one off answer, everything is great, but if the solution is to be made into proper enterprise ready code, the entire notebook must be transcribed into truly disciplined code. (as an aside, this process is massively difficult because data scientists often don't understand the principles of object oriented programming, and the programmer doesn't understand the principles specific to the data science objective the code was written to solve.)

      I expect to use Jupyter a lot more frequently in the coming years, but I fear it will feel like a huge step back in terms of the things that computer scientists have solved, that data scientists are ignoring.

      --
      Which has more power: the hammer, or the anvil?
    2. Re:Huge Notebook fan. by TechyImmigrant · · Score: 2

      I wrote a book in latex, with lots of python code generating data and gnuplot and matplotlib generating pictures. This worked well (try doing a 426 page mathy book in word) and it was all text files, so source control and offsite backup via git worked well.

      Would you recommend Jupyter for that kind of thing. Would the output always look like a paper - or could you make it work for technical book writing to eliminate some of the scripting and hand integration done with raw latex?

      My experiments were not encouraging, but it's entirely possible I was just clueless when trying.

      --
      I should use this sig to advertise my book ISBN-13 : 978-1501515132.
    3. Re:Huge Notebook fan. by 0100010001010011 · · Score: 2

      It looks exactly like it would on paper, but if you plan on printing it I don't know if it handles the page break stuff. It does make a great PDF.

      I know that you can integrate LaTeX templates.

      The nice part about it is you don't have to deal with 99% of LaTeX and can just focus on writing the equations.

      There's a free online way to try it: https://jupyter.org/try

    4. Re:Huge Notebook fan. by Hognoxious · · Score: 2

      Most are mechanical or electrical, and while they are subject matter experts their code leaves a lot to be desired.

      A bit like programmers, then?

      --
      Confucius say, "Find worm in apple - bad. Find half a worm - worse."
    5. Re:Huge Notebook fan. by Lab+Rat+Jason · · Score: 2

      I think you misunderstood a few things. On my first point I said there is no INTEGRATION with source control. In Visual Studio, you can commit and check out directly in your IDE. In Eclipse, you can commit and check out directly in your IDE. In Pycharm you can commit and check out directly in your IDE. In Jupyter you can not (at least I'm not aware of how).

      On your second comment, there are many ways to test that the code compiles and runs correctly, but that will not guarantee that it is right. I, like you, tend to frequently reset the kernel to ensure everything is good, but my concern isn't for me so much as it is for a group of scientists, who are scientists first, and data scientists second, and in some cases data scientists first, but computer scientists second. I'd love to see a build server and unit test capability in Jupyter, but I know I'm reaching on that one... asking data scientists to write using object oriented principles and unit tests is akin to asking them to write poetry in greek.

      On the third point, it is again, an area that I don't have an issue with so much as it is an area that a group of data scientists has an issue with. If I want to promote rapid deployment of data science packages to the organization, I need an easier way to move code from spitball POC to enterprise-ready, load tested, environmentally robust code. I can't begin to tell you how many times I've seen people try to deploy code to a server, only to enter into dependency hell because project A wants a package that is incompatible with project B. Now I've got to circle back and containerize all these piddly applications, and once that's done, I don't have a way to push the containers back to maintainers (because they're scientists dammit! They don't have time to learn about containers!). There aren't enough hours left in my life (and I'm not THAT old) to do all of this myself. I need as many people as possible to up their game doing data science to make any real progress. That means taking content matter experts, and training them with data science skills. Because that's way easier than taking someone with data science skills and teaching them the subject matter. That means I need a development environment that is easy enough for them, and also provides enterprise software development capability.

      Anyway, I guess all I'm saying is that I pray for the day that this ecosystem is as mature as the ones I'm coming from because a lot of the problems that have long been solved in computer science are now being repeated. Those who fail to learn from history are doomed to repeat it.

      --
      Which has more power: the hammer, or the anvil?
    6. Re:Huge Notebook fan. by 0100010001010011 · · Score: 2

      You can execute shell commands from within a notebook with !

      !git commit -am "Commit message"

  3. Literate Programming by Anonymous Coward · · Score: 2, Informative

    Knuth tried to teach us to do this decades ago, but nobody listened.

  4. Because Jupyter aligned with Mars? by jfdavis668 · · Score: 3, Funny

    Then we will analyze what guides the planets and understand what steers the stars.

  5. A rare sort of development in the software world.. by Junta · · Score: 3, Interesting

    Jupyter is something that is relatively unique, useful in its field, and *not* crammed down the throats of people for whom it isn't really relevant.

    I applaud the way that project is executed, adopted, and evangelized as being on point and solidly executed...

    --
    XML is like violence. If it doesn't solve the problem, use more.
  6. Terrabytes/day is not much for astronomy by Anonymous Coward · · Score: 2, Interesting

    In early nineties, my experiments used to generate 5 terabytes of data a day (limited by ability to store and post process. the telescopes had capability of generating much larger data. In fact in late 70s and early 80s, it was common to generate terrabytes of data a day in radio astronomy VLBI experiments. These were stored on regular video cassettes (7 GB/tape). A single experiment will use anywhere from 5-25 recorders simultaneously 24 hrs a day.

  7. Re: Couldn't be STUPYDR by epine · · Score: 2

    I get it. You print out the suspect word with your 3D printer, and then you use your sterling hand to trace around the printed artifact for bumps, hollows, ridges, descenders, and cisterns—which you simultaneously compare and contrast to the irregular outlines of your Mario Kart Kamikazi Kukmumbr as braced by your other fleshy mitt.

    Phrenology, with benefits.

  8. Literate programming by jma05 · · Score: 3, Interesting

    For decades, we talked about Knuth's literate programming. Jupyter is finally an open source tool that made it usable for everyone.
    There is no better way to explain the use of a library than making a Jupyter notebook available.

    Most of my Python use lately is for one-off analytics with heavy libraries. Jupyter suits this workflow very well.
    IPython already has decent hooks for IDEs (PyCharm, Spyder), but I hope this gets even better.

  9. It's Matlab in python clothing by goombah99 · · Score: 3, Interesting

    I love jupyternotebooks. But it's matlab. Well a broken inferior matlab. I do like python syntax better than matlab but that's just a sugar.

    The upcoming Jupiterlab is a slavishly copy of the matlab ide.

    It reminds me of how Linux desktop managers were always copying the last generation of windows.

    I'm not complaining! I use mint and it owes a lot to windows too.

    Mint however is actually superior to windows now.

    But look at something like staroffice libre office. Ow... the pain. It's like a bad ms office 5, except you can only use it if you have thumbtacks in your shoes. They copied everything that was bad just so it was the same.

    Jupiter is really nice and I use it in preference to matlab because it's so portable and I can use other python packages. But unless you used matlab you may not realize it's just a fast follower of ideas already tested out by matlab

    --
    Some drink at the fountain of knowledge. Others just gargle.