Slashdot Mirror

← Back to Stories (view on slashdot.org)

How a Computer Game Is Reinventing the Science of Expertise

Posted by Soulskill on from the so-many-volunteers dept.

An anonymous reader writes "Cognitive scientists at Simon Fraser University and UCSD are beginning to use StarCraft 2 replays to study the development of expertise and the cognitive mechanisms of multitasking. Unlike similar expertise studies in chess that consider roughly a dozen players, these studies include thousands of players of all skill levels — providing an unprecedented amount of data on how players move from 'chumps to champions.'"

60 comments

  1. any other studies? by wierd_w · · Score: 5, Funny

    Like a study on the mass exodus of players doing ladder play after the koreans find the tournament?

  2. wow by Pierre+Bezukhov · · Score: 1

    this is really cool. that is all.

  3. Development of Shortcuts by Anonymous Coward · · Score: 4, Insightful

    Ninety percent of the players in this tournament are not superhuman multitaskers. After watching enough of their first-person video streams, you see that most players can't react to novel situations. They just learn "build an army before the X minute mark", or "counterattack with fast ground units when his army moves out," executing the same limited skill set game after game. What this study will probably show is the rapid development of mechanical skills from low-to-mid level play, followed by the gradual acquisition of timings and strategic instincts.

    1. Re:Development of Shortcuts by wisnoskij · · Score: 4, Interesting

      Well of course a novel play would slow them down. The reason they are so fast is because they no longer thing about the normal stuff, it is ingrained like a champion chess player. But that does not mean that they are unable to cope with the never seen before.

      --
      Troll is not a replacement for I disagree.
    2. Re:Development of Shortcuts by khallow · · Score: 2

      And that would be good to know. It's worth noting here that the basic learning curve model is that in a doubling of the time that someone does an activity, then their efficient gets a certain fraction better. This was used to describe improvements in manufacture.

      For example, one such rule of thumb is that doubling the number of widgets produced in a manufacturing process, results in a 5-15% reduction in the marginal cost per widget. I gather this was used like Moore's Law as a sort of self-fulfilling prediction. If the process didn't improve by the desired amount then you spent more to achieve the target improvement (or you lowered expectations by reducing the rate of improvement).

      But such theory doesn't take into account qualitative differences. As you imply in your example above, in Starcraft, there's three crude, abstract regimes of play. Logistics is the management of production and resource harvesting, tactics is short term movement and use of military resources particularly in fights, while strategy is the long term planning of the game. You might also make a four category of physiological/mental adaptations to the peculiar Starcraft environment such as image recognition, muscle memory, acting while under stress, adaptive behaviors (such as scanning), etc.

      This gives the researchers ways to refine ideas about learning. It may well be that in the future, that developers of a process will not only be able to set reasonable expectations for improvement of the process, but also relatively simple metrics that highlight the most likely areas for improvement.

    3. Re:Development of Shortcuts by Hentes · · Score: 1

      Yeah, these C&C-type RTSes are about micromanagement, not strategy.

    4. Re:Development of Shortcuts by AaronLS · · Score: 1

      This is similar to chess in that players learn to identify certain counter strategies to other strategies. Another analogy would be like card counting in black jack, in that rather than do a full calculation of probabilities in your head, you've abstracted that process to a set of techniques. Of course these techniques are not as accurate or reliable, but in a realtime game like Starcraft, you don't have time for formulating a detailed war plan.

    5. Re:Development of Shortcuts by Anonymous Coward · · Score: 0

      Back in the early days of competitive FPS tournaments I read an interview with the winner of some big 1 on 1 competition where he explained why he thought he was such a good player: He was continually predicting the other players actions. Most players don't do that until they can see the other person, or if they do it's at a really basic level of "he might come from that direction, or if he went the other way through the map he might come from this direction a few seconds later". What he was doing was continually running a narrative of the best possible choices the other player could be making, "if he goes that way he won't be able to grab health, but I just nailed him before he retreated". As he learned the kind of choices his better opponents would make he could predict their actions perfectly wherever they were on the map. It'd seem like ESP or hax, you would see him fire a rocket at nothing and then the other player would run through a door right into his rocket. The only way you could beat him was to play crazy, you had to be as unpredictable as possible.

    6. Re:Development of Shortcuts by Anonymous Coward · · Score: 0

      It's not that difficult - basically the best way to play a FPS is to constantly run along the shortest paths between armor, health, weapons and weapon recharges. Then knowing what direction a player is running in would give an indication of what they are running low on.

    7. Re:Development of Shortcuts by Reapy · · Score: 1

      I think most action games have a meta game at the upper level. Just like you said, a good player playing this predictive one, would be thinking, oh, well he expects me to grab the health since that is my best choice, I'm going to go grab the shotgun over here instead. 95% of the time getting the health is the right move, but when two masters of any game square off, you have to make the 'wrong' move some of the times. Just have to be sure the wrong move isn't suicide.

      It's silly but I used to play at mount & blade warband a lot, and I felt like the skill climb culminated in playing like a beginner. A lot of times if you were used to playing good players, if you found yourself squared off against someone just a step above total newbie, you could often get yourself killed because they would act suicidally. In your mind from experience, it is known that attacking at that moment will net them being hit.

      Many times you fake out good players, you get in a position where they have no way of attacking you back, they must defend, and you fake them, attacking another way. The beginner just swings, no regard for the perilous situation he is in. Where it succeeds is if the advanced player is already reacting as though there should be no possible way he would think to even attack in that situation, he does, and hits the advanced player.

      I think a real master will know just when to act like a beginner, doing things that appear completely suicidal, yet somehow coming out unscathed.

      Things like this probably have to do with a game moving fast as well. As they article said the SC2 players with more APM's tend to win. Having a fast reaction time, being primed to react immediately to situations sooner than your opponent will usually trump amazing predictive abilities more often than not. I guess it is deadly when the guy with amazing reaction time also has the ability to run the predictions too.

  4. OT: Simon Fraser University by Anonymous Coward · · Score: 0

    ...has been used as the new Tollan homeworld on Stargate: SG1 and Caprica's Riverwalk shopping district on Battlestar Galactica, among numerous others.

  5. Re:Old News by Anonymous Coward · · Score: 0

    Herp derp.

  6. As a fellow cognitive scientist... by Anonymous Coward · · Score: 4, Informative

    Let me be the first to explain "why Starcraft 2?"

    The answer lies in the oft-cited measure of player skill at the game: actions per minute. This is an unprecedented numerical measure of expertise that lends itself well to the study of "expertise" -- a term which means something different in the study of the brain than it does to the everyday person. Expertise is nothing less than a figurative rewiring of your brain in order to better excel at a chosen repetitive task. You can check out Wikipedia if you want to read more about it.

    1. Re:As a fellow cognitive scientist... by ahabswhale · · Score: 4, Interesting

      As a daily SC2 ladder player, I can definitely state that APM has little to do with skill. It's a useful metric to determine how well a skilled player can execute his strategy but that's it. It doesn't, in any way, indicate whether a player can dynamically adjust his play to beat his opponent. It just means that if he can come up with a sufficient strategy, that he will probably be able to make it happen (assuming, of course, his opponent doesn't throw a wrench into the works). So, it scares me that anyone doing research in this field would put so much weight into APM.

      --
      Are agnostics skeptical of unicorns too?
    2. Re:As a fellow cognitive scientist... by AaronLS · · Score: 1

      Yeh, I've watched videos where a player during any kind of idle time will click on and off things just to keep their APM up...

    3. Re:As a fellow cognitive scientist... by Anonymous Coward · · Score: 3, Informative

      Actually, APM has a lot to do with skill. But it's important to differentiate between raw APM and effective APM.

      Raw APM is literally how many clicks and keystrokes you execute per minute. Effective APM is how many of those keystrokes were meaningful (repeatedly clicking the same move command over and over has no effect on the game and is thus not meaningful). The trick here is that the game obviously measures raw APM, and it's difficult for the game to discern exactly what separates meaningful clicks from meaningless ones, although Blizzard recently implemented an algorithm to try. If someone is just spamming the keyboard and mouse, they may have a high raw APM, but their effective APM is still low.

      Anyway, effective APM measures your ability to multitask and effectively control your army and base. It doesn't have anything to do with high-level strategy, except that more APM allows you to execute on strategies more effectively; indeed, there are many strategies that are more or less impossible without having a high APM. It's also not a perfect measure, since it doesn't calculate how intelligent each move was in the context of the given strategy. But, it's better than nothing, and you can't play at a pro or even semi-pro level without being able to get your APM high.

    4. Re:As a fellow cognitive scientist... by Anonymous Coward · · Score: 1

      Actually, if you RTFA, you'd see that from analyzing 3500 replay files the study has established that players who make more actions per minute, and whose actions are more widely distributed across the map, do tend to win.

    5. Re:As a fellow cognitive scientist... by ahabswhale · · Score: 1

      All really good players have a high APM but not all people with high APM are really good players. Thus, it's a shit statistic.

      --
      Are agnostics skeptical of unicorns too?
    6. Re:As a fellow cognitive scientist... by Vaphell · · Score: 1

      it's about warming the hands up and keeping the intensity high, so you can easily pull off these 300apm with no filler in decisive moments of the game. Some spam apm to enlarge the e-peen but they are not pros and more often than not it makes them play much worse. It's not a measure of skill but more like a measure of physical potential, what you do with that potential is an entirely different story.

    7. Re:As a fellow cognitive scientist... by Anonymous Coward · · Score: 0

      No, this is wrong. If you assume a player isn't just pointlessly spamming, then APM is a valuable statistic. But, it's more useful for individuals to track their own progress (if your APM is increasing and you haven't been deliberating gaming the statistic, then you're probably getting faster), than it is to directly compare two players.

    8. Re:As a fellow cognitive scientist... by Anonymous Coward · · Score: 0

      This is also like saying, "All good runningbacks are fast, but not all fast people are good runningbacks. Thus, running speed is a shit statistic." APM is just one component of player skill. You could say the same thing about ANY aspect of play: "All good sc2 players have good high-level strategy, but not all good high-level strategists are good sc2 players." A person can be fast with the mouse and still be terrible (or more commonly, pointlessly spamming and still be terrible), of course.

    9. Re:As a fellow cognitive scientist... by Derkec · · Score: 1

      I think the "distributed across the map" part is key. In beginner / intermediate play, a really key ability is to actually remember to keep building stuff while scouting or fighting. Can you throw down that building on time while also looking at your opponent's base and gleaning useful information there in an early scout? In an early skirmish, can you micro well enough to gain a minor advantage while also still building workers for the long term economy that actually matters? In the midgame, it's worse, you have building workers, troops, scouting, managing supply, map positioning, upgrades, watching the mini-map for counters/drops, etc, etc, etc all going on at the same time in different places of the map and screen.

      90% of the players in the world can't really do these things consistently well because the require you to take care of a lot of background work while actively focused on the exciting stuff. That's genuinely hard and you get better at it incrementally at best.

    10. Re:As a fellow cognitive scientist... by Anonymous Coward · · Score: 0

      You're right. I like to help people from my local CSL on strategy and improvement since I'm no longer eligible to play and find that Gold level players have tighter openings and stronger fundamental mechanics at the start of the game than I do (High Diamond/Low Masters depending on what season you look at). The problem comes when the game starts branching out. Its very easy for them to manage 1 or 2 bases almost flawlessly, but once their army is half way across the map their actions are spread out and slow down or are less meaningful. Once the attack starts their base sits almost idle or they only do certain actions while concentrating on a finite section of the map. As skill increases that range of concentration widens as they're able to take in more information and process it. The reason I feel that I've progressed further than these players who have many more games under their belt and a stronger feel for the "metagame" is that I have better cognative abilities to watch and predict things than they do.

      tl;dr - tunnel vision.

  7. Too focused? by Anonymous Coward · · Score: 0

    StarCraft seems to be a good case as it certainly is a game that requires significant multitasking.

    However, there are other things which work in starcraft that don't seem to help with relating expertise in StarCraft with other areas of study. Use of hotkeys, unit management, unit production styles and map exploration techniques. The data from this study would only seem to help with determining experts at StarCraft or Real time strategy games.

  8. What you do is get every gamer to wear a helmet. by lexsird · · Score: 1

    So that you can scan all kinds of brain functions while this happens. It would help with brain function mapping as well, which we seriously need to know and understand. Brain surgery needs to go beyond just cutting hunks of meat, it needs to be about helping reroute neural networks in organic units, to include humans.

    There is so much that we still don't understand and frankly it's annoying.

    --
    Take the Red Pill.
  9. Starcraft Races by TehNoobTrumpet · · Score: 1

    I like how they describe the Protoss as 'Photosynthetic Aliens' compared to the Terran as 'Humans' and the Zerg as 'Insectoid Aliens'
    I mean, I guess that is part of the lore that Protoss photosynthesize, but definitely not the first thing that comes to mind.
    More like 'Super Advanced Aliens' vs 'Super Primal Hive Insect Aliens"

    1. Re:Starcraft Races by gl4ss · · Score: 1

      it's just humans, horde and elves.

      nothing more!

      --
      world was created 5 seconds before this post as it is.
    2. Re:Starcraft Races by bckrispi · · Score: 1

      What about Undead? (You insensitive clod!) :)

      --
      Xenon, where's my money? -Borno
  10. Re:What you do is get every gamer to wear a helmet by masternerdguy · · Score: 2

    The problem is the number of cells in the brain. It took years to map the brain of a damn worm, your brain is orders of magnitude larger and has even more orders of magnitude in connections. Besides, I don't think we have a scanner that can read activity on such a small scale yet. You'd only be able to map their brain after killing them and putting them under a scope.

    --
    To offset political mods, replace Flamebait with Insightful.
  11. Day traders start taking notes by earthsmurf · · Score: 2

    RTS games are the most applicable to a wide range of real world systems & strategies than say FPS games or any other type of game. It's not only emergency systems that it compares to, but also at a high level to many different business types & strategies as well. Techniques such as knowing when to scout, expand, and attack all require precise timing just like in the business world, maybe even for the day trader. I wonder if successful day traders have a high APM?

    --
    - Anything that can be put in a nutshell should remain there.
  12. Last lizard guy. by carpefishus · · Score: 1

    When the lizard guy comes to recruit you, don't go. It's not Hollywood out there.

    --
    Facts take all of the premium out of arm waving - T. Reynolds
  13. Re:What you do is get every gamer to wear a helmet by lexsird · · Score: 1

    We need general information, not microscopic at the moment. That's like shooting a mosquito with a bazooka. I am sure we can monitor plenty of brain activity and map some data. With large numbers like this and them all in the same type of brain function format, it's ideal for extrapolating observational data.

    Think of it as watching brain functions like one would watch car traffic from above. We hope we could be tree top level, but we are doing good to just watch from orbit. If we can bring it down closer we can gather more data. Eventually we need to be "boots on the ground" with our levels of not only observation but operation, but that seems like a bit down the line. Lets knock out what we can do now while we are here.

    --
    Take the Red Pill.
  14. Re:OT: Simon Fraser University by Garridan · · Score: 1

    That is NOT the reason I go to SFU. It's... one of a few reasons.

  15. Re:What you do is get every gamer to wear a helmet by AaronLS · · Score: 1

    I saw a documentary where a brain surgeon was scanning a musicians brain while they were doing brain surgery and had the musician hum tunes while they mapped general locations so that they'd avoid hacking out those pieces while removing a tumor. They literally just had letters written on little pieces of paper and were just letting the wetness of his exposed brain hold them on :O

  16. APM is a distorted metric, many actions useless by perpenso · · Score: 5, Insightful

    As a fellow cognitive scientist let me be the first to explain "why Starcraft 2?" The answer lies in the oft-cited measure of player skill at the game: actions per minute. This is an unprecedented numerical measure of expertise that lends itself well to the study of "expertise" -- a term which means something different in the study of the brain than it does to the everyday person. Expertise is nothing less than a figurative rewiring of your brain in order to better excel at a chosen repetitive task.

    APM is a distorted metric. It does not distinguish between a meaningful action, a redundant action, a nervous "twitch" (i.e. multiple clicks rather than one), etc.

    Furthermore it contains an additional distortion. Since it is a metric that players are evaluated by, and/or used in silly "pissing contests", it can be intentionally distorted. Why click on that point on the ground once when you can click on it five time rapidly? APM focused players often are manically clicking on empty ground issuing no unit orders when they have nothing to do for a second or two, they have rewired their brain to have them do "something" even if there is nothing useful to be done.

    For the programmers reading along, Think of APM as the LOC (Lines of Code) of the Starcraft world. Both metrics can be meaningful in an idealized setting, but such is not the setting of most real world events.

    1. Re:APM is a distorted metric, many actions useless by Anonymous Coward · · Score: 1, Interesting

      You're half-right. Many of those "useless" actions actually do have a purpose: to keep up your pacing. It's easier to keep up clicking maniacally the entire game than to repeatedly accelerate and decelerate as the demands in the game go up and down. That's why you see people spamming clicks at the beginning of the game. Sure, by themselves clicking on the minerals a whole bunch is pointless, but it acts as a warm-up for the rest of the game.

  17. Their data should be used to make an expert AI! by Dr.+Spork · · Score: 1

    It's awesome that there are potentially millions of SC2 replays available for data mining. They're surprisingly tiny files. And I think the really cool research project would be to data mine those files for strategies that the AI could master on its own. Because I suck at SC2, I don't really want to go out and embarrass myself in games against other people, so I often play against the game's AI. I also - reflexively - pause the game sometimes, which would be really impolite against a human opponent. But the game's AI is outrageously predictable and linear in how it approaches the game. I'd like to hope that once researchers discover what makes a starcraft player good, they can translate this to an AI expert system and make it play like a good player. Like there are computer chess tournaments, wouldn't it be fun to have SC2 AI tournaments? We'd learn a lot about AI in the process, probably more than we ever did through writing chess programs.

    1. Re:Their data should be used to make an expert AI! by Anonymous Coward · · Score: 0

      There is no randomized damage in starcraft. That means with equal reaction time (hardware) and equal resources, there will always be a single ideal sequence for the AI to execute against another AI. They'd improve for a few years and then stalemate, it'd become more about guessing / learning the enemy AI than anything to do with the game. That'd still be pretty interesting. But I bet if you let that develop and mature for a few years you got an AI that played pretty damned stupid against a human, being tuned to out-guess other machines.

    2. Re:Their data should be used to make an expert AI! by matthiasvegh · · Score: 1

      While I'm probably out of my depth here, surely in SC2 an AIs performance could be measured in some fashion. I don't know if theres such a concept as a score in SC2, but an evolutionary algorithm pitted against humans, with the aforementioned score as a health function may breed an ai algorithm which is almost ideal. Could the SC2 playing residents here shed some insight into this?

    3. Re:Their data should be used to make an expert AI! by Anonymous Coward · · Score: 0

      If you want a better AI to play against, go here http://sc2.nibbits.com/assets/green-tea-ai/

    4. Re:Their data should be used to make an expert AI! by TomOTooleNZ · · Score: 1

      I don't have it to hand to refer to, but didn't Galactic Civilizations (or maybe it was GC 2) do something like this? Use records of players' gameplay to update the AI and then send out updates?

      --
      as any fule kno
  18. The Science of Expertise? by tsa · · Score: 1

    The title is confusing. Expertise is not a science. You (hopefully) get expertise when you do research, as the article explains.

    --

    -- Cheers!

    1. Re:The Science of Expertise? by yndrd1984 · · Score: 1

      The title is confusing. Expertise is not a science.

      Right. But when you talk about how the scientific investigation of X is done, you can refer to that subject as 'the science of X'. Not a common way to phrase it, but it is correct. Would you rather they called it 'expertisology'?

    2. Re:The Science of Expertise? by Paul1969 · · Score: 1

      Would you rather they called it 'expertisology'?

      Oh yes, please. Can we?

    3. Re:The Science of Expertise? by gl4ss · · Score: 1

      maybe it's re-inventing pedagogy. maybe. but doing something to learn to do it is hardly revolutionary, except if you compare it to "humans as robots" way of teaching people to do things, which was hot shit at beginning of last century(for industry jobs where you just repeated the same task, then it made sense to see how the best one doing it did it and then copy those movements and teach them to everyone..).

      --
      world was created 5 seconds before this post as it is.
  19. Re:What you do is get every gamer to wear a helmet by lexsird · · Score: 1

    Amazing, if I do say so. I see we need to engineer some better tools for our scientists to use to study brains with. I guess I need more biology classes as well. I should have started this 40 years ago.

    Here's what I think of, an array set about the skull to pick up any kinds of RF, something sensitive, then some IR to check any kinds of thermal levels. What other kind of stuff could we monitor and use to project a mathematical graphing of this? Break out the fancy graphing calculator for that.

    We need someone in the industry to finance this of course and to produce the helmets. Sony? lol...Nintendo? Microsoft? I'm not a billionaire yet, so I can't just whip them out at one of my many factories and run over to the tournament and pass them out. Think of the applications you could work then! Wow, you could just put a helmet on if you had all this information mapped. Humans work well with biofeedback equipment as it is. Think of the poor handicapped people who could then control about anything you hook up into their control. They could run factories.

    I think it would shed lots of new data onto the situation, and give us some retrospect to some of our "human reverse engineering" we have done plenty of in the past. Seriously, we have mapped the human genome... let's get this done.

    --
    Take the Red Pill.
  20. Re:champion chess player by Anonymous Coward · · Score: 1

    I'll reply to you.

    Seems to me this is a severely flawed discussion, well into Apples & Oranges territory.

    Let me talk from a modest knowledge of the chess side. Chess does not normally reward "actions per minute" except the subset of Blitz events. Yes, GMs have higher "throughput of variations" than amateurs, but generally so I've heard is that a "mere master calculates, the GM 'knows'." The GM simply doesn't even look at the weak moves. Then all that analytic firepower gets focused on grade A moves.

    There are Many GMs upset with faster time controls. They operate best when their intuition takes a minute and a quarter to sift through the position and get the right "idea" then check it for 30 sec and make a move. That's a totally different metric than this "actions per minute" stuff.

    Last is Sample Size. Any study with only 12 players is a joke. That's a funding / management problem. I bet 1000 chess players would love $100 to be in a study.

    Posting AC but sayin' --TaoPhoenix

  21. "micromanagement, not strategy" by Nova+Express · · Score: 2

    “Amateurs talk about strategy, professionals talk about logistics.” - Carl von Clausewitz

    Now, what's Starcraft about again?

    --
    Lawrence Person (lawrencepersonh@gmailh.com (remove all "h"s to mail)

    http://www.lawrenceperson.com/

    1. Re:"micromanagement, not strategy" by Tubal-Cain · · Score: 1

      Economics

    2. Re:"micromanagement, not strategy" by Shinobi · · Score: 1

      And yet both the amateur and the professional is right in regards to Clausewitz's statement. For any strategy in the real world, 90-95% of the strategy is logistics. No, they are not separate.

      As for Starcraft, it's a bare minimum abstractions tactics, all focusing on micromanagement, which is a deathtrap in real life. People who were good in Brood Wars etc who later went into officer academy or became NCO's turned out to be really bad leaders and tacticians. Many of them have aspergers or OCD, and when faced with an environment where they have to factor in morale, lead times, the fact that micro-management is detrimental etc they couldn't cope with it.

    3. Re:"micromanagement, not strategy" by Hentes · · Score: 1

      "There are in Europe many good generals, but they see too many things at once. I see one thing, namely the enemy's main body. I try to crush it, confident that secondary matters will then settle themselves." - Napoleon

      He was a guy famous for not caring about supply lines.

    4. Re:"micromanagement, not strategy" by Derkec · · Score: 3, Insightful

      How did that work out for him in Russia?

    5. Re:"micromanagement, not strategy" by ultranova · · Score: 4, Informative

      How did that work out for him in Russia?

      He reached Moscow. So I'd say it worked for him far better than most "conquer Russia" strategies have.

      --

      Forget magic. Any technology distinguishable from divine power is insufficiently advanced.

  22. No, StarCraft, not wow. by forkfail · · Score: 0

    nt

    --
    Check your premises.
  23. Re:What you do is get every gamer to wear a helmet by Alsee · · Score: 1

    You'd only be able to map their brain after killing them and putting them under a scope.

    Obviously that's phase two. We couldn't exactly gather the two data sets the other way around, now could we?

    -

    --
    - - You can't take something off the Internet! That's like trying to take pee out of a swimming pool.
  24. Seems to over come O.C.D. anyway by thirdwikidotorg · · Score: 1

    I saw a programme on the American chess master who died in Iceland who played chess games. I forget his name.. But he seemd to be preoccupied with real world events that shaped his belief system ... too much. I think that keeping it Virtual allows for people to engage on a fun level without the need for revealing one's self. It seems to overcome the O.C.D. anyway.

  25. gucci handbags,louis vuitton handbags by cheap4guccibags · · Score: 1

    http://www.cheap4gucci-bags.com/ gucci handbags,louis vuitton handbags,prada handbags.coach handbags,burberry handbags

    --
    http://www.cheap4gucci-bags.com gucci handbags,louis vuitton handbags