Slashdot Mirror


User: photon317

photon317's activity in the archive.

Stories
0
Comments
1,300
First seen
Last seen
Profile
(view on slashdot.org)

Comments · 1,300

  1. Re:PostgreSQL vs MySQL on Sun Eyes PostgreSQL · · Score: 4, Informative


    The biggest one that has made a difference in my life lately:

    Table Partitioning:
    PostgreSQL > MySQL; Mainline PostgreSQL has table partitioning as of 8.1-beta, by leveraging inheritance (Postgres is an Object-Relational Database).

    Queries on the aggregate of the partitions are directed at the parent table, and optimized to only look into appropriate sub-table by checking CHECK constraints of the sub-table against the query WHERE clause.

    Basically, you do it like this (contrived, but related to how I'm using them at the moment):

    MyBigFatTable stores timestamped data from a bunch of a machines at regular intervals, keying off of the machine id and the timestamp of the data:

    CREATE TABLE MyBigFatTable (
        machineid INTEGER REFERENCES machines(machineid),
        stamp TIMESTAMP,
        data_x FLOAT,
        data_y FLOAT,
        [... lots more data fields ...],
        PRIMARY KEY (machineid, stamp)
    );

    Your problem is, the table size grows and grows and grows unbounded, and database operations continue to get slower and slower (inserts, updates, and selects) as the table grows. You have a policy to expire the data after a month which limits the maximum growth, but this in turn requires lots of deletes happening all the time, which again hurts performance.

    The inheritance-based partitioning solution is to leave that table definition as it is, and also define:

    CREATE TABLE MyBigFatTable-2005-10-05 (
        PRIMARY KEY (machineid, stamp),
        FOREIGN KEY (machineid) REFERENCES machines(machineid),
        CHECK ( stamp >= '2005-10-05 00:00' AND stamp '2005-10-06 00:00')
    ) INHERITS MyBigFatTable;

    As you can see, the column definitions are inherited, but you must re-specify the PK/FK stuff. The added check clause says that only data from Oct 10, 2005 is valid in this subtable.

    You set up a maintenance script to create your new time-based tables ahead of time (say once a day create tables for the next day), and you do your data INSERTs into the specific subtable (you know the timestamp of the data you're inserting, so you can generate the appropriate table name from that (MyBigFatTable-2005-10-05).

    You run your SELECTs against the original MyBigFatTable just as you did before. It automatically includes any rows from its child tables. Further, if your SELECT's WHERE-clause was constraining a query to a specific time-range, only those children of MyBigFatTable whose CHECK constraint indicates they could possibly have relevant data are checked.

    And as for the problem of expiring data and the delete traffic you had before? You simply drop the old child tables with "DROP TABLE" from a maintenance script when they're a month old - no DELETEs neccesary.

  2. Re:xargs and for loops on What's Your Command Line Judo? · · Score: 1


    Just because you think globbing on the commandline is stupid (and I don't neccesarily agree with you, but I haven't really thought through all the ramifications of ydoing it your way yet) doesn't make it go away. That being the environment we're dealing with on standard unix shells, xargs is a big win for dealing with that environment. Write us a new shell and a new set of the standard commandline binaries that does everything your way, or deal with the status quo like the rest of us :)

  3. Re:xargs and for loops on What's Your Command Line Judo? · · Score: 2, Interesting


    Oh a couple more good simple ones popped into my head for beginners:

    Use "&&" between commands, and the second command will only execute if the first command succeeded. For example:

    make && make install

    That will only run the "make install" if the "make" succeeded (which is kinda contrived, since for most packages a simple "make install" would have done the same all on it's own).

    And also, did you know that you can group commands with parentheses, and you can pipe into parenthesized groups of commands? Piping into a group of commands will send the pipe data to the first command in the group that actually wants it. You can copy a directory over from one place to another using tar all in one line like so:

    tar -cf - . | (cd /otherdir/; tar -xvBpf -)

    The two tar commands are simply piping a tar-format file between them without using the disk, and the second tar runs with it's current directory set to /otherdir/.

  4. xargs and for loops on What's Your Command Line Judo? · · Score: 4, Informative

    There's millions of tricks, but if I had to a couple simple powerful techniques that anyone should learn that doesn't know them already, it would be xargs and commandline "for" loops.

    xargs takes whatever is piped into it, and executes a command with those things as arguments. It can do it all at once, or it can break them up in chunks, or it can execute your command once per input. Consider:

    rm -f `find . -name "*.o"`

    This normally works fine, and will forcibly remove all .o files anywhere underneath the current directory. However, if there are too many .o files to fit on a single commandline, it will barf with "argument list too long" or some such sounding error. The xargs way to do this would be:

    find . -name "*.o" | xargs -n 50 rm -f

    Which will execute a seperate "rm -f" for each chunk of 50 filenames. Take a look at the "-i" mode as well, read the whole man page. It's a great little peice of glue.

    On to for loops. You've seen them in sh/ksh/bash shellscripts like so:

    for fn in *.c
    do
        echo Sending $fn ...
        rcp $fn remotehost:/tmp/
    done

    You can of course do this straight from the commandline, which is indispensable for complex looped operations. To do it all in one line, you just have to get the semicolons in the right place. Just remember there's a semicolon before the "do", but not immediately after it:

    for fn in *.c; do echo Sending $fn ...;rcp $fn remotehost:/tmp/;done

  5. Re:Wow on Trigonometry Redefined without Sines And Cosines · · Score: 1


    FP hardware doesn't do integration, or anything else that complex, in a single instruction either. Whether the FPU does it, or you hand-code integer instructions to do what the FPU unit was doing, either way at the bottom layer it is being completed as simple binary arithmetic.

  6. Re:Sounds good to me on Mobile Phones Locked By DMCA · · Score: 1


    The problem is that you might purchase a locked phone initially, and there are companies out there which, for a small fee, will forcibly unlock your phone for you when you decide to switch carriers. (Or if you have the technical knowledge and means, you can do it yourself). They're arguing that unlocking a phone which was purchased as a locked phone is a violation of the DMCA.

  7. Re:This hasn't been my experience on High-Performance Linux Clustering · · Score: 2, Informative


    MOSIX is really more of a halfway-point between a traditional cluster and a "single system image" sort of cluster. Unfortunately, some aspects of clustered computing are still extremely difficult to abstract away into an ssi type of implementation. I had hoped over the years that the MOSIX work would get folded in with mainstream Linux's NUMA scheduling and memory allocation, essentially treating non-local cpu and memory resources (other nodes) like a second layer of NUMA with even less connectivity than local NUMA nodes have. Throw in a truly distributed redundant filesystem (ala the google filesystem that we don't know all the details about), and we could really begin to approach the concept of turning large-scale local and even distributed clusters into truer single system images. But the kinds of fundamental work that needs to be done to get these things rolling hasn't even started, so I don't expect it to happen anytime soon. Aside from all that clustering and kernel work, there would have to be some evolutionary changes in how we write code, and in the languages we write it in, in order to smoothly take advantage of the dynamic availability and locality of resources easily.

    For now, your best bet is to construct your HPC linux cluster as a high-speed network of interconnected but independant Linux machines using a network topology that suits the class of problems you face and how easily the problem can be broken into loosely coupled peices, and then code the "clustering" aspect into your application code itself, usually working off of libraries like MPI.

  8. Nifty, but.. on DIY Electronic Paper Display · · Score: 2, Insightful


    I'd hardly call it a "DIY" kit at a cost of $3,000. And it's not shipping for at least another month. And judging by thier screenshots, even simple fonts look fairly crappy at this resolution and only 4-level grayscale. If it were $150 I'd consider it for a home project. If it were $1,000 for the devkit with a promised volume price of under $100, I might consider developing a product with it, if I already had a great idea that I was fairly confident of. But for $3,000, who's buying this first-gen technology devkit with unknown technological future and unknown (but probably high given the devkit cost) pricing?

  9. Re:Hmmm... on Owning Your Own IP at a Company? · · Score: 1


    Also, my understanding is that under many employment contract terms (and in some US States, even just by default if you are a salaried employee), the company owns all of your intellectual output, period. An example: If you work as a Java programmer writing JSP software for an online auction company by day, and one saturday evening at home on your own time, you conceive and construct a new type of sprinkler system for your lawn, and patent it, the software firm can legally take your patent away from you in court if they want it.

  10. Re:Wow on Trigonometry Redefined without Sines And Cosines · · Score: 1


    I agree, but at the same time, most programming disciplines (even when coding from "scratch" and not using library functions of any kind) don't involve any math that's really more complicated than basic algebra in computer language form (which means we can do things like iterative operations, sums, and branching the equasion based on intermediate results in intuitive ways).

    The very fact that there really are no intrinsic trigonometric or "calculus" operators in common assembly languages (exluding the unneccesary FPU and SIMD/MIMD acceleration units) speaks volumes to this. On an 80386 (with no 80387), one can still write software which essentially "solves" the problems that can only be solved by trig and calculus. This proves (as mathematicians know I'm sure) that trig and calculus are reducible to complex algebraic constructs using tests/branches, iterators, etc.

    I would argue that the "language" of calculus as taught in maths class is somehow unneccesarily complex and unintuitive, and could probably be taught to more programmers if it were taught to them in the form of patterns of algebraic constructs that can be pretty much 1:1 turned into basic code that doesn't use library functions or FPU stuff.

    Of course it would still be the same thing mathematically, but it's all about the form in which it is expressed.

  11. Re:rsync on Subversion as Automatic Software Upgrade Service? · · Score: 1


    I tend to agree with the parent. You might want to do version control on your software releases with subversion, but ultimately you should check out the new stable copy you want everyone upgraded to and then distribute it via other means, like rsync. rsync is particularly a good choice because it will only send the minimum amount of data neccesary to get the job done efficiently.

  12. Re:I'm confused on Stolen U.C. Berkeley Laptop Recovered · · Score: 1


    It's illegal to steal a laptop, and if you're caught possessing and/or selling stolen goods, that's automatically probable cause for arrest on charges of theft. Arrest is only the first step in a long process which may or may not lead to the conviction of you or someone else you ratted out on criminal charges.

  13. Re:Photoshop is the *wrong* tool on A Simple Tool for Tracking Switch Ports? · · Score: 1


    Or if you want something free, you could try "dia" or "kivio".

  14. Re:Down with TLDs! on CentralNic Enables uk.com Wildcard DNS · · Score: 1


    No, the drawback is that DNS was designed as a heirarchical distributed information lookup system. It handles an insane amount of queries per day in a distributed fashion, and has performed phenomenally well all things considered.

    Part of the magic of this heirarchy is that the tree is smaller at the top and wider at the bottom. It just flat-out wouldn't scale or work at all if every domain-name owner today was using a TLD instead of something underneath some part of the heirarchy with fewer names.

    The heirarchy is there for technical reasons more than aesthetic appeal.

  15. Re:How Does OnStar send back info from car to "bas on GMC to Begin Remotely Scanning Cars for Trouble · · Score: 1


    I believe OnStar communicates via cellular networks, but I'm not entirely sure.

  16. There's some BS on the site on The Electrocharger...Any Day Now? · · Score: 3, Interesting

    There is the remote possibility that it can have some benefits. The basic idea is vaguely sound. However, they're fond of making wildly inaccurate statements on their website, so I'd take any specific claims with a grain of salt.

    My favorite FAQ is:


    Q. Is the Electrocharger(TM) a performance upgrade?

    A. Yes, it decreases your 0-60 time by a minimum of 3 seconds and sheds at least 3 seconds off your 1/4 mile time, over stock vehicle performance.


    This is of course an outrageous statement to make without qualifications. My current vehicle does 0-60 in about 4.2 seconds. There is no way in hell that thing is going to give me a 1.2 second 0-60. Perhaps on a car that goes 0-60 much much slower, then maybe you could shave 3 seconds, but even then that sounds like an awful lot.
  17. Your language doesn't support it on Why Does Current Clustering Require Recoding? · · Score: 3, Informative


    The only way you'll have source code that compiles and runs unmodified on architectures of widely varying parallelism efficiently is for the language itself to know about parallelism, and make it the compiler's (and even runtime-linker and kernel's) job to parallelize your code for you. An inherently parallel language would have ways for you to specify in your source code what can and cannot be executed in parallel, and what code absolutely depends on the serial execution of some previous code. Even then, we're really only talking about the SMP case. When you start involving network latencies and bandwidth restrictions, the decisions on when and how to parallelize become more challenging for the compiler/runtime, possibly requiring either more intelligence on its part and/or more meta-information in your source code.

    Until you write code in a language like that, you can never expect to write code in a single-threaded mindset and then have it just magically take advantage of a parallel environment.

  18. Re:Why not just make electricity? on Making Ice Without Electricity · · Score: 1


    He wasn't correct however. He simply glossed over the glaring problem the parent posters were pointing out. You have to blow air into the thing really hard to make it work. How do you plan on blowing air into this simple device really hard continuously for days without involving some other more complex machine that probably relies on electricity or some other power source?

  19. Re:Suns have been 64 bit for a while now... on Sun Unveils 64-bit Server Line · · Score: 4, Insightful


    That's a great reverse-reverse-psychology troll or something, but it's hardly insightful. I have always been a Sun supporter over the decade or so I've been working with Solaris on Sparc. I have always said that they were the best *nix thing going out there, compared to their competition. But I was also always a fan of Linux where it was applicable.

    But the time came (some time ago now) to admit that Sun has in fact missed the boat on Linux. What made it especially frustrating was that, of all of the commercial *nix vendors, Sun was in the best position to capitalize on the Linux wave. They were already all about developing and promoting open standards (think NFS and NIS back in the day, among many others). They were already the best non-free platform to build and use open source software on. Hell starting with Solaris 8 they were shipping a good deal of open source software with stock Solaris. But some idiot(s) in charge of the company completely lacked the vision to make it happen. I can only imagine how much better a position Sun would be in (and how much better off all consumers of *nix would be) if Sun had re-centered themselves around Linux kernels going forward back in the late 90's or even 2001-ish. They could've turned their kernel engineering teams to work on Linux on Ultrasparc (and Opteron), and could've brought a lot of scalability and other enhancements with them to the Linux kernel in general to boot.

    Even now that Sun has started to turn the corner on Linux from their previous stances (which were to ignore it, and then to marginalize it as a toy), their stance still smells a lot like, "Sure, run linux on our Sun-branded but otherwise whitebox-like and overpriced x86 and x86_64 hardware, but only for crappy unimportant edge devices. Leave all the real computing to a real operating system like Solaris." Meanwhile smart companies are working out strategies to transition off of the last remaining Ultrasparc behemoths they have left in the corner of the datacenter while the majority of their real computing is already happening on Linux today. Average not-so-smart companies will be doing this in a few years.

    I don't hate Sun, and I don't think they're Evil. But I think someone fell asleep at the wheel there and completely failed to take advantage of the Linux wave like Sun should have. If anything, I feel sad for them, it's tragic to watch a great company go down like this. They could still turn it around, but I don't have much faith anymore that they will.

  20. Re:You may, good sir! on ESR Gets Job Offer From Microsoft · · Score: 2, Insightful


    He's an intelligent and thoughtful guy, and certainly worth whatever fame he's managed to acquire. It's not that the man doesn't have skills, it's just that the sheer amount of geekiness (the kind that gets you laughed at) outweighs what valuable skills he possesses. Nobody would make fun of Linus for being as geeky as ESR, for example, because his skill is extraordinary enough to justify it.

  21. Re:It should be noted on Unpatched Firefox Flaw May Expose Users · · Score: 1


    In many cases, a bug which causes a crash when triggered with inappropriately long data turns out to be a bug which can be exploited to execute arbitrary code if the data is carefully crafted to do so. Your test merely reconfirms the basics of this bug. In all likelyhood, the guy can run arbitrary code via this bug if he's claiming he's done it.

  22. Re:A few steps back? on GNOME 2.12 Released · · Score: 1


    My 2.10 I've being using from stable gentoo never crashes. Perhaps you should try "stable" gentoo, or scale back your CFLAGS :)

  23. Re:Very Funny on 6.8GHz 1TB RAM and 2TB HDD Laptop? · · Score: 2, Informative


    Yeah, but an actual 64-bit processor is not capable using 16 exabytes of memory. AFAIK, both x86-64 platforms (amd opteron/athlon64 and intel's ripoffs) are limited in silicon to addressing 1TB of physical RAM. Operating system support can of course push this number lower. The kernel shipping with SuSE Enterprise Linux on an Opteron can address up to 512GB.

  24. laid down the rules? not hardly on Review: Dungeon Siege II · · Score: 2, Insightful
    the original Diablo laid down many of the rules for the hack and slash adventure genre


    Diablo, and every other game remotely similar to it, really drew their inspiration in terms of computer game heritage from the rogue-like text adventures, of which Nethack is the most shining example. While they've tacked on nifty stuff like shiny graphics and networked multiplayer, they have still completely and utterly failed to capture even 5% of the depth and complexity of gameplay that Nethack has enjoyed for years.
  25. Re:it's != its on Saturn Moon Continues to Delight and Baffle · · Score: 1


    A complete set of American English grammar written in BNF would be painfully complicated and long. It would brilliantly show how inconsistent and ill-designed the language is.