Even if the fibers are small, they may invoke kernel calls which take a long time to complete (for example a large I/O operation). How do you ensure that other fibers can continue their work in such cases?
For example, to write a large file over the network, the scheduler write small piece of data at a time. And all socket are on non-blocking mode. So if the read/write operation would block, the OS just raise an "would block" error, and the scheduler use epoll to check back later.
To call synchronous function, there are solutions like forking the operation into on an other process. And there are different solutions for the process-communication : pipes, sockets, hlnet (our home made protocol used by our distributed DB), rest, soap...
Further, the smaller your fibers get, the more overhead you introduce for context-switching, right? Because I can imagine that after execution of each fiber you need to do some scheduling (even if the current fiber remains active). Is this context-switching faster than the kernel can do it?
Finally, I'm interested in what makes scheduling for the web such a specialized task, but that may be related to the previous question.
By "specialized", I mean it's better that fibers are scheduled by the application-scheduler itself, that knows the logic of the application and the purpose of each fibers, rather than the OS-scheduler. About context-switching, there is the small context-switching between fibers inside the main-thread, and the bigger context-switching between this mean thread and the others thread of the OS. So it really depends here on what we are trying to compare (and if it's comparable).
Don't forget that the compiler transform your code to a CPS representation. So in practice, your OPA function is "cut" into many very very small fibers. So, even if we are talking about non-preemptive fibers, the high-level user function can be stopped by the scheduler at any time to take car of new I/O events.
It's the scheduler job to balance all those small fibers (that belong to several high-level functions) to optimize the latency level. And the web-specific scheduler does this job better than the generic OS-scheduler.
Hi StripedCow, I'm one of the guys who wrote the OPA scheduler so I'll try to answer your scheduler-related questions.
It uses fibers (co-operative light threads). The compiler transforms your code to a CPS representation, so our scheduler can balance the pool of computation stuff and I/O operations (we use epoll on Linux, kqueue on Mac OS). The developer can explicitly push asynchronous tasks with a simple Scheduler.push https://opalang.org/resources/doc/index.html#scheduler.opa.html/!/value_stdlib.core.rpc.core.Scheduler.push
OPA doesn't yet offer to the developer the control of the priorities.
So as you can understand, it doesn't rely on the OS-scheduler, there is only one main OS-thread for all computations and I/O clients (our scheduler does the job). A common question is how to use all cores of a multi-cores machine? Well, just launch the app one more time and use your load balancer (or use our cloud tool).
Compiled their 76-line hello_chat.opa into a 30M executable
In those 30Mo you have just everything (the web server, the database engine, the scheduler, the page engine, the session engine, your webpages source, css, the generated js, etc) ! It's a standalone binary that just works on your server without having to install anything more. Did you check the size of the LAMP / Java / Ruby packages?
I would love to see some eCom or other complex sites written in opa to convince me that this is a viable alternative to Ruby, Java, PHP, etc.
MLstate did launch an eCom-like site : http://jetleague.com/
And I just check the binary on our server: 17Mo (yes, I guess it has been upx-ed)!
We did have a working Windows version and we may have it again. Now OPA is open source, maybe there are some volunteers to help us:) ?
Note that we will soon launch something that allows *everyone* to try OPA. Keep in touch!
Slashdot Mirror
Even if the fibers are small, they may invoke kernel calls which take a long time to complete (for example a large I/O operation). How do you ensure that other fibers can continue their work in such cases?
For example, to write a large file over the network, the scheduler write small piece of data at a time. And all socket are on non-blocking mode. So if the read/write operation would block, the OS just raise an "would block" error, and the scheduler use epoll to check back later.
To call synchronous function, there are solutions like forking the operation into on an other process. And there are different solutions for the process-communication : pipes, sockets, hlnet (our home made protocol used by our distributed DB), rest, soap...
Further, the smaller your fibers get, the more overhead you introduce for context-switching, right? Because I can imagine that after execution of each fiber you need to do some scheduling (even if the current fiber remains active). Is this context-switching faster than the kernel can do it?
Finally, I'm interested in what makes scheduling for the web such a specialized task, but that may be related to the previous question.
By "specialized", I mean it's better that fibers are scheduled by the application-scheduler itself, that knows the logic of the application and the purpose of each fibers, rather than the OS-scheduler. About context-switching, there is the small context-switching between fibers inside the main-thread, and the bigger context-switching between this mean thread and the others thread of the OS. So it really depends here on what we are trying to compare (and if it's comparable).
The ultimate web language with no file uploads?
https://mlstate.com/forum/feedback_on_opa/feedback/1#5
This thread is one year old. Now, we do have an upload component in our stdlib : https://opalang.org/resources/doc/index.html#upload.opa.html/!/value_stdlib.upload.Upload
I hope my message above answers this question too. http://developers.slashdot.org/comments.pl?sid=2401364&cid=37234382
Don't forget that the compiler transform your code to a CPS representation. So in practice, your OPA function is "cut" into many very very small fibers. So, even if we are talking about non-preemptive fibers, the high-level user function can be stopped by the scheduler at any time to take car of new I/O events.
It's the scheduler job to balance all those small fibers (that belong to several high-level functions) to optimize the latency level. And the web-specific scheduler does this job better than the generic OS-scheduler.
Hi StripedCow, I'm one of the guys who wrote the OPA scheduler so I'll try to answer your scheduler-related questions. It uses fibers (co-operative light threads). The compiler transforms your code to a CPS representation, so our scheduler can balance the pool of computation stuff and I/O operations (we use epoll on Linux, kqueue on Mac OS). The developer can explicitly push asynchronous tasks with a simple Scheduler.push https://opalang.org/resources/doc/index.html#scheduler.opa.html/!/value_stdlib.core.rpc.core.Scheduler.push
OPA doesn't yet offer to the developer the control of the priorities.
So as you can understand, it doesn't rely on the OS-scheduler, there is only one main OS-thread for all computations and I/O clients (our scheduler does the job). A common question is how to use all cores of a multi-cores machine? Well, just launch the app one more time and use your load balancer (or use our cloud tool).
The code is open-source, scheduler included ;)
Compiled their 76-line hello_chat.opa into a 30M executable
In those 30Mo you have just everything (the web server, the database engine, the scheduler, the page engine, the session engine, your webpages source, css, the generated js, etc) ! It's a standalone binary that just works on your server without having to install anything more. Did you check the size of the LAMP / Java / Ruby packages?
I would love to see some eCom or other complex sites written in opa to convince me that this is a viable alternative to Ruby, Java, PHP, etc.
MLstate did launch an eCom-like site : http://jetleague.com/ And I just check the binary on our server: 17Mo (yes, I guess it has been upx-ed)!
No windows port? seriously?
We did have a working Windows version and we may have it again. Now OPA is open source, maybe there are some volunteers to help us :) ?
Note that we will soon launch something that allows *everyone* to try OPA. Keep in touch!