lukestein’s avatarlukestein’s Twitter Archive—№ 17,297

                1. At the end of the semester, many of us talking about AI and assessment. I’m in my second year attempting in-class exams where students are allowed to use LLMs. x.com/lukestein/status/1920283805333545267 Quick example of an exam question that’s been interesting to see. (1/
                  Permalink On twitter.com Twitter logo lukestein’s avatar ❤️ 7 Favorites Mood +2 🙂
              1. …in reply to @lukestein
                Between the close of the US equity markets on April 2 and April 3, 2025, big US tariffs were announced and many stock prices suffered large declines. Would you expect a relationship between ticker length and the size of stocks’ decline around the tariff announcement?
                oh my god twitter doesn’t include alt text from images in their API
                Permalink On twitter.com Twitter logo lukestein’s avatar ❤️ 1 Favorite Mood -1 🙁
            1. …in reply to @lukestein
              We start by investigating one-letter ticker symbols. We've written code similar to this in the classroom together. Some students do it like this. Many others wind up with 100 lines of LLM-written stuff and try to pip install beautifulsoup or whatever
              oh my god twitter doesn’t include alt text from images in their API
              Permalink On twitter.com Twitter logo lukestein’s avatar ❤️ 1 Favorite Mood +2 🙂
          1. …in reply to @lukestein
            GPT 4o confidently asserts there are four such companies. If pressed, it will admit that there are maybe five. Or six. Or seven. If we tell it all ten, it will apologize for the oversight. chatgpt.com/share/681bf97b-3490-8002-ae30-c110724c9d4c
            Permalink On twitter.com Twitter logo lukestein’s avatar ❤️ 2 Favorites Mood -1 🙁
        1. …in reply to @lukestein
          Claude 3.7 Sonnet finds 9. It is more circumspect ("reflects my knowledge as of October 2024, and there may have been changes to the S&P 500 composition since then.") claude.ai/share/926f4d42-aced-4d4f-8b36-08768c7d40ca
          oh my god twitter doesn’t include alt text from images in their APIoh my god twitter doesn’t include alt text from images in their APIoh my god twitter doesn’t include alt text from images in their API
          Permalink On twitter.com Twitter logo lukestein’s avatar ❤️ 1 Favorite Mood 0
      1. …in reply to @lukestein
        Perplexity says seven, but if pressed agrees maybe it's nine. No wait, not nine; it's definitely seven. perplexity.ai/search/how-many-single-letter-ticker-4gILUkTERDaoWbMG93sQlw
        oh my god twitter doesn’t include alt text from images in their API
        Permalink On twitter.com Twitter logo lukestein’s avatar ❤️ 1 Favorite Mood 0
    1. …in reply to @lukestein
      (bw, chatGPT o3 nails it with three minutes of thinking. That model was released publicly two weeks after this midterm exam.) chatgpt.com/share/681cb11c-6f08-8002-b3c3-c13909174b2d
      Permalink On twitter.com Twitter logo lukestein’s avatar ❤️ 1 Favorite Mood 0
  1. …in reply to @lukestein
    Of course, it's more interesting to think about whether/why there might be a relationship between ticker length and tariff announcement effects. There are many potential interesting mechanisms. If students think about some—even if an LLM proposed them—great!
    On twitter.com Twitter logo lukestein’s avatar ❤️ 1 Favorite Mood +4 🙂
    1. …in reply to @lukestein
      I heard lots of different things across students, and note different LLM models/prompts (or random draws) gave different suggestions. Won't post student work, but here's some LLM output.
      oh my god twitter doesn’t include alt text from images in their APIoh my god twitter doesn’t include alt text from images in their APIoh my god twitter doesn’t include alt text from images in their API
      Permalink On twitter.com Twitter logo lukestein’s avatar ❤️ 1 Favorite Mood 0
      1. …in reply to @lukestein
        Plausible hypotheses go in different directions! And in talking about how they might assess mechanisms, strong student answers made reference to specific data (e.g., around international trade, corporate supply chains, liquidity) and analytical techniques we discussed in class.
        Permalink On twitter.com Twitter logo lukestein’s avatar ❤️ 1 Favorite Mood +2 🙂
        1. …in reply to @lukestein
          Thanks for making it this far. I'm procrastinating grading, but here's a question from yesterday's final exam. In the course of *writing* this question, I learned more than I expected about the inconsistency of FX rate data (both availability and levels) across data sources.
          oh my god twitter doesn’t include alt text from images in their APIoh my god twitter doesn’t include alt text from images in their APIoh my god twitter doesn’t include alt text from images in their API
          Permalink On twitter.com Twitter logo lukestein’s avatar ❤️ 1 Favorite Mood +2 🙂
          1. …in reply to @lukestein
            I also learned I should ask some expert if there's a reason to expect more extreme JPYUSD fluctuations around the 8th and 9ths of the month (LLMs speculate this may be plumbing issues around clearing in Japanese markets during GFC). Excited to see what my students can teach me!
            oh my god twitter doesn’t include alt text from images in their API
            Permalink On twitter.com Twitter logo lukestein’s avatar ❤️ 1 Favorite Mood +3 🙂