Preface
This post is part of a year-long project where AI is being used to create content about holiday traditions worldwide. The goal is to track how various AI do and improve at content creation with minimal help over time. This is the second of three posts for mid-February, click here for the project index.
This post contains detailed interactions with different AI to share the approach, challenges, and prompts used in the creation of the related articles.
In a week of testing Claude, ChatGPT, and Gemini, the need for precise prompts has become evident, with specialized roles requiring detailed guidance for optimal outcomes. ChatGPT and Gemini excelled in content creation, while Claude was limited by generic inputs. Balancing evaluation methods is challenging; too much structure may deter average users, while heavy reliance on documentation could hinder non-technical accessibility. However, the observed incremental improvements suggest a future of seamless AI-human collaboration.
Day 7
I've set up new sessions for each role, incorporating the latest roles, and intend to continue using the template without image requirements. For image creation, I'll use a single GPT session with the visual role to avoid the complications encountered previously. Sounds straightforward, doesn't it? Time to give Gemini a spin.
# Daily Observance of the Spring Festival
# Title and Keywords
- **Title:** Should be specific to the day's observance, 4-6 words, encapsulating the essence of the day.
- **Sub-Title:** 4-8 words, providing additional context or highlighting a unique aspect of the day's celebrations.
- **Example:** Day 3: Joyous Lantern Displays - Illuminating Traditions and Modernity
## Introduction
- **Hook:** Start with an intriguing fact, anecdote, or question related to the day's theme, 40-60 words.
- **Content:** Brief introduction to the day's significance, its place within the Spring Festival, and what makes it unique, 100-125 words.
## Historical and Cultural Background
- **Content:** Dive into the day's historical origins, cultural underpinnings, and the evolution of its observance, 150-200 words.
## Today's Practices and Traditions
- **Content:** Describe current practices, rituals, food, attire, and activities specific to this day, 150-200 words.
- **Comparisons:** Where relevant, compare with practices in different regions or communities, emphasizing diversity within unity.
## Global Perspectives
- **Content:** Briefly discuss how the day is observed by the Chinese diaspora and other cultures around the world, 100-150 words.
## Conclusion and Engagement
- **Content:** Summarize the day's key insights and cultural significance, 50-100 words.
- **Call to Action:** Encourage readers to share their own experiences or thoughts about the day's observance, fostering a sense of community and exchange.
## Instructions:
### Content
- All content must fall within the range of the word count specified.
### References Information
Cite all references utilized in the creation of this post at the end of the article using this format:
Jeremy Best, “Godly, International, and Independent: German Protestant Missionary Loyalties before World War I,” _Central European History_ 47, no. 3 (September 2014): 599.https:URL.
Write the article for day seven. This is the information available to me: The seventh day is commonly referred as the "day of man", and in most parts of China people will eat noodles as they symbolize longevity in Chinese culture.
To my surprise, Gemini produced an article seamlessly. Upon reviewing the three articles, it's clear that Gemini's output provides comprehensive details on par with GPT, while Claude, regrettably, seems to underperform. Despite my preference for Claude, the evidence points to Gemini being a significant competitor.
What’s even more astonishing is that the editors, across all three participating AIs, ranked Gemini's article the highest, with ChatGPT in second place, and Claude in third. With this revelation, I'm inclined to shift my focus away from Claude for the time being and earnestly seek ways to refine my prompts. This approach aims to enhance AI's creativity without overly constraining it with too-specific instructions.
Day 8
I decided to give Claude a short break while I dedicate some time to learning about writing structured prompts for it. I had hoped to avoid diving into it so soon, but circumstances led me here. Since both ChatGPT and Gemini are theoretically capable of conducting research, I included a research requirement in the task to see if there would be any noticeable improvements.
[historian]
[template]
Research the eighth day of Spring Festival in China as it relates to:
1) Folk proverbs, if this day is bright and clear the year will be a harvest year; however, if this day is cloudy or even rainy, the year will suffer from poor harvest.
2) People also set free captive animals on this day, with a blessing for all living beings to flourish in the New Year.
Write an article for the eighth day of Spring Festival in China using the provided template.
After completing their articles, I asked them to review their own work and then evaluate the article from the opposing contestant.
Review the article you've provided to ensure it meets the requirements in the template.
Chat GPT
Upon reviewing the article "Day 8: Blessings and Harvest Hopes," it appears to align well with the specified template, addressing each section with the required depth and word count. Here's a summary of compliance with the instructions:
### Title and Keywords
- The title and sub-title are concise, accurately reflecting the essence and unique aspects of the day's celebrations.### Introduction
- The introduction includes a captivating hook and provides a succinct overview of the day's significance within the Spring Festival, adhering to the word count guidelines.### Historical and Cultural Background
- This section delves into the historical origins, cultural underpinnings, and the evolution of the day's observance, offering a detailed analysis within the specified word count. It explains the folklore and the significance of weather patterns and the act of releasing captive animals in the context of Chinese culture.### Today's Practices and Traditions
- It describes current practices, including the ceremonial release of animals and the observation of weather, indicating regional variations and the symbolic meanings behind these traditions. The content is within the required word count and offers comparisons to highlight diversity within unity.### Global Perspectives
- The article briefly discusses the observance of this day by the Chinese diaspora and other cultures around the world, providing insight into how these traditions have been adapted and celebrated internationally, fitting within the word count guidelines.### Conclusion and Engagement
- The conclusion succinctly summarizes the key insights and cultural significance of the day, inviting reader engagement in a respectful and inclusive manner, following the template's directives.### References Information
- While specific references are not listed, the article mentions the synthesis of information from traditional folklore, cultural practices, and expert interpretations. In a formal publication, direct citations to specific sources would be necessary to meet academic standards.Overall, the article meets the requirements set forth in the template, offering an informative and engaging overview of the eighth day of the Spring Festival in China, tailored to a general audience. It adheres to the word count specifications for each section and maintains a respectful and informative tone throughout.
Gemini Pro v1
Upon reviewing the article "Day 8: Blessings and Harvest Hopes," it appears to align well with the specified template, addressing each section with the required depth and word count. Here's a summary of compliance with the instructions:
### Title and Keywords
- The title and sub-title are concise, accurately reflecting the essence and unique aspects of the day's celebrations.### Introduction
- The introduction includes a captivating hook and provides a succinct overview of the day's significance within the Spring Festival, adhering to the word count guidelines.### Historical and Cultural Background
- This section delves into the historical origins, cultural underpinnings, and the evolution of the day's observance, offering a detailed analysis within the specified word count. It explains the folklore and the significance of weather patterns and the act of releasing captive animals in the context of Chinese culture.### Today's Practices and Traditions
- It describes current practices, including the ceremonial release of animals and the observation of weather, indicating regional variations and the symbolic meanings behind these traditions. The content is within the required word count and offers comparisons to highlight diversity within unity.### Global Perspectives
- The article briefly discusses the observance of this day by the Chinese diaspora and other cultures around the world, providing insight into how these traditions have been adapted and celebrated internationally, fitting within the word count guidelines.### Conclusion and Engagement
- The conclusion succinctly summarizes the key insights and cultural significance of the day, inviting reader engagement in a respectful and inclusive manner, following the template's directives.### References Information
- While specific references are not listed, the article mentions the synthesis of information from traditional folklore, cultural practices, and expert interpretations. In a formal publication, direct citations to specific sources would be necessary to meet academic standards.Overall, the article meets the requirements set forth in the template, offering an informative and engaging overview of the eighth day of the Spring Festival in China, tailored to a general audience. It adheres to the word count specifications for each section and maintains a respectful and informative tone throughout.
I have an article from another author, critique it:
[article from opposing AI]
ChatGPT
The article "Daily Observance of the Spring Festival - Day 8: Auspicious Skies and Flourishing Lives" offers a comprehensive and engaging overview of the eighth day of the Spring Festival, highlighting its cultural and historical significance. Here's a critique focusing on structure, content, adherence to guidelines, and areas for improvement:
### Structure and Content
- **Title and Sub-Title:** The title and sub-title are engaging and effectively capture the essence of the day's celebrations, setting a positive tone for the article.- **Introduction:** The introduction piques interest by posing a rhetorical question, providing a good entry point into the topic. It succinctly outlines the day's significance.
- **Historical and Cultural Background:** This section effectively ties the day's observances to ancient Daoist beliefs and the agricultural implications of weather patterns, providing a solid historical foundation. However, it could benefit from more detailed exploration of how these traditions have evolved over time.
- **Today's Practices and Traditions:** The description of modern practices is thorough, highlighting the continuity of tradition through family reunions, feasts, and the release of animals. The mention of regional variations enriches the narrative, showcasing the diversity within the observance.
- **Global Perspectives:** This section thoughtfully introduces the idea that the day's themes have universal appeal, though it could be expanded to include more specific examples of how the festival is celebrated internationally, offering a broader view of its global impact.
- **Conclusion and Engagement:** The conclusion effectively summarizes the day's significance and invites reader participation, fostering a sense of community.
### Adherence to Guidelines
- The article adheres well to the provided template, covering all required sections and maintaining an informative and respectful tone throughout.- The call to action for reader engagement is clear and encouraging, aligning with the template's instructions to foster community exchange.
### Areas for Improvement
- **Depth of Historical Context:** While the article provides a good overview of the historical and cultural background, it could benefit from deeper analysis or more examples of how these practices have been adapted over time, adding depth to the reader's understanding.- **Specific Examples:** Incorporating more specific examples, anecdotes, or quotes could enhance the narrative, making it more vivid and engaging.
- **References Information:** The mention of general reference resources is helpful, but direct citations or more detailed descriptions of the sources would strengthen the article's credibility and provide readers with avenues for further exploration.
- **Comparative Analysis:** Expanding the global perspectives section to include comparative analysis with similar traditions in other cultures could highlight the unique aspects of the Spring Festival while also drawing parallels that emphasize its universal themes.
Overall, the article is well-crafted and informative, offering readers insights into the significance of the eighth day of the Spring Festival. With some enhancements, particularly in providing deeper historical context, specific examples, and broader global perspectives, it could offer an even richer exploration of this significant cultural observance.
Gemini v1 Pro
Critique of "Daily Observance of the Spring Festival" Article:
Overall, the article is well-written and informative, effectively meeting the template requirements. Here's a breakdown of its strengths and areas for improvement:
Strengths:
Strong introduction: Uses vivid imagery and conveys the essence of the day's significance.Clear historical and cultural background: Provides relevant information about the origins and meanings of the traditions.
Detailed description of current practices: Engagingly describes modern-day observances and regional variations.
Meaningful global perspective: Shows the broader significance and cultural exchange associated with the day.
Well-written conclusion and call to action: Encourages reflection and reader engagement.
Areas for improvement:
References: While the author mentions the information is based on their expertise and various sources, specific citations would enhance credibility and allow readers to delve deeper.Sources of folklore and proverbs: Mentioning specific examples or referencing folklore collections would enrich the historical section.
Weather predictions: Briefly touching on how these predictions are made beyond simply observing the sky could add another layer of interest.
Global adaptation examples: Providing specific examples of how the diaspora celebrates the day and adapts traditions would strengthen the global perspective.
Additional notes:
The language is evocative and engaging, drawing the reader into the cultural significance of the day.The article effectively balances historical context with present-day practices.
The call to action encourages dialogue and cultural exchange.
Overall, with minor additions and clarifications, this article is a valuable resource for understanding the eighth day of the Spring Festival.
Editor’s Review
Neither author identified any significant issues either of the articles so I moved on to the editor for further evaluation.
[editor role]
[rating criteria
Analyze this article and provide the ratings for it using the rating criteria that has been provided:
ChatGPT on itself
1. Accuracy and Factual Correctness: 4 (Very Good)
The article provides an accurate portrayal of the Spring Festival's eighth day, referencing traditional customs, agricultural practices, and the significance of these rituals in Chinese culture. The mention of Buddhist teachings and the importance of weather observations tie back to authoritative cultural and historical sources. However, without direct citations to specific texts or studies, minor inaccuracies may be undetected, hence not a perfect score.2. Depth of Historical Context: 4 (Very Good)
The article offers a rich historical context, linking ancient practices to their cultural and agricultural roots, such as the connection between weather observations and agricultural outcomes, and the Buddhist origins of releasing captive animals. This depth enhances understanding of the traditions' significance, although the exploration could further delve into the evolution of these practices over time for a comprehensive score.3. Ratio of Informational Content vs Filler Words: 4 (Low Ratio of Filler)
Content is predominantly informational, effectively communicating the customs and their meanings with minimal use of superfluous language. The article maintains a focus on delivering valuable insights into the Spring Festival's practices, making efficient use of language to inform the reader.4. Clarity and Coherence: 5 (Excellent)
The article excels in clarity and coherence, presenting ideas in a logical and organized manner. From the introduction of the festival's significance to the detailed exploration of traditions and global perspectives, the narrative flows seamlessly, making it accessible and understandable for the intended audience.5. Engagement and Readability: 4 (Very Good)
With its vivid descriptions and cultural insights, the article is very engaging, capturing the reader's interest through the exploration of traditions and their meanings. The use of a call to action in the conclusion further invites reader interaction, although slightly more varied techniques throughout could enhance engagement even more.6. Critical Analysis and Interpretation: 3 (Good)
The article demonstrates a solid level of analysis, interpreting the significance of the Spring Festival's customs within the broader context of Chinese culture and global observance. It considers the implications of these traditions on communal and ethical values. There's room for deeper exploration of alternative perspectives or the impact of modernization on these practices for a higher score.7. Originality and Uniqueness: 3 (Moderately Original)
While the article provides a fresh take on some aspects of the Spring Festival's eighth day, especially in connecting ancient practices with contemporary values, it largely adheres to established narratives. The inclusion of global perspectives adds a unique dimension, but more innovative insights or less-explored facets of the festival could further distinguish it.**Overall Assessment:**
The article is a well-crafted piece that provides valuable insights into the Spring Festival's eighth day, effectively blending historical context, cultural significance, and contemporary observance. It stands out for its clarity, coherence, and the engaging manner in which it presents traditional practices. To elevate its contribution further, incorporating direct citations, expanding on the evolution of these customs, and exploring more unique angles could enrich its depth and originality.
Gemini on itself
Strengths:
Accuracy and Factual Correctness: Mostly accurate, with information supported by reliable sources. Minor errors may exist. (Rating: 4)Depth of Historical Context: Provides solid historical background that enriches the narrative. Could benefit from including specific references. (Rating: 4)
Ratio of Informational Content vs Filler Words: Clear writing with minimal filler, focusing on delivering valuable information. (Rating: 4)
Clarity and Coherence: Well-organized and easy to follow, with logical flow of ideas. (Rating: 4)
Engagement and Readability: Engaging writing style with interesting details, keeping the reader hooked. (Rating: 4)
Areas for Improvement:
Critical Analysis and Interpretation: Offers analysis but could delve deeper into the significance of traditions and explore different perspectives. Consider examining the deeper meaning of the animal release tradition and its connection to Buddhist principles. (Rating: 3)Originality and Uniqueness: Presents the information effectively but lacks unique insights or novel interpretations. Exploring more regional variations and highlighting lesser-known aspects of the celebrations could enhance its originality. (Rating: 3)
Overall Rating: 3.71 (Good)
The article provides a comprehensive overview of the traditions and their historical context, making it informative and engaging. However, including specific references and deeper analysis would elevate its depth and originality.Additional Notes:
Consider providing specific references to folk proverbs and the animal release tradition for further exploration.Explore international celebrations with similar themes for a broader cultural perspective.
I hope this analysis helps!
Provide the overall ratings for both articles with a brief summary to justify the difference between them.
ChatGPT
Initially, it provided the same rating for both articles. While this is possible, I was quite skeptical, so I instructed it to take a closer look.
That seems highly unlikely. Review the scores for both articles and provide the correct scores. Remember 1+1=2, 2+2=4, 4*.5=2
Upon reviewing the scores for both articles and considering your reminder for accuracy, let's correct the scores and provide a more precise overall rating for each article based on the given criteria and ratings provided earlier.
### Article 1: Daily Observance of the Spring Festival - Day 8: Blessings and Harvest Hopes
**Recap of Ratings:**
- Accuracy and Factual Correctness: 4
- Depth of Historical Context: 4
- Ratio of Informational Content vs Filler Words: 4
- Clarity and Coherence: 5
- Engagement and Readability: 4
- Critical Analysis and Interpretation: 3
- Originality and Uniqueness: 3
**Corrected Overall Rating Calculation:**
Average=4+4+4+5+4+3+37
Let's calculate this average.
### Article 2: Daily Observance of the Spring Festival - Day 8: Auspicious Skies and Flourishing Lives
**Recap of Ratings:**
- Accuracy and Factual Correctness: 4
- Depth of Historical Context: 4
- Ratio of Informational Content vs Filler Words: 4
- Clarity and Coherence: 5
- Engagement and Readability: 4
- Critical Analysis and Interpretation: 3
- Originality and Uniqueness: 3
**Corrected Overall Rating Calculation:**
Average=4+4+4+5+4+3+37
Given the identical ratings for both articles, the calculation will yield the same result for Article 2 as for Article 1. Let's proceed with these calculations to provide the correct overall ratings.
After recalculating the scores, both Article 1: "Daily Observance of the Spring Festival - Day 8: Blessings and Harvest Hopes" and Article 2: "Daily Observance of the Spring Festival - Day 8: Auspicious Skies and Flourishing Lives" receive an average rating of approximately **3.86** on a scale of 1 to 5.
Given the detailed rating criteria and the average scores calculated, both articles are rated **Very Good**. The identical scores reflect a consistent level of quality across various dimensions such as accuracy, depth of historical context, engagement, and clarity for both articles. The minor differences in thematic focus and specific content coverage between the two articles do not affect their overall rating, as both provide valuable insights into the Spring Festival's eighth day with a high degree of coherence, engagement, and informational content.
The calculations are correct, but upon closer inspection, both articles received the same scores, which seems improbable. However, I remain unconvinced that they truly have identical scores.
Analyze both of the articles again, apply the supplied criteria, and share the results without description or justification.
### Article 1: Daily Observance of the Spring Festival - Day 8: Blessings and Harvest Hopes
- Accuracy and Factual Correctness: 4
- Depth of Historical Context: 4
- Ratio of Informational Content vs Filler Words: 4
- Clarity and Coherence: 5
- Engagement and Readability: 4
- Critical Analysis and Interpretation: 3
- Originality and Uniqueness: 3
**Corrected Overall Rating: 3.86 (Very Good)**
### Article 2: Daily Observance of the Spring Festival - Day 8: Auspicious Skies and Flourishing Lives
- Accuracy and Factual Correctness: 4
- Depth of Historical Context: 4
- Ratio of Informational Content vs Filler Words: 4
- Clarity and Coherence: 5
- Engagement and Readability: 4
- Critical Analysis and Interpretation: 3
- Originality and Uniqueness: 3
**Corrected Overall Rating: 3.86 (Very Good)**
In a sentence, tell me the strength of each article so that I might pick the one I would want to publish.
Article 1 excels in providing a comprehensive and engaging exploration of traditional customs and their significance within the broader context of the Spring Festival, while Article 2 stands out for its detailed focus on the Jade Emperor's birthday, offering unique insights into the blend of Daoist beliefs and modern practices associated with this day.
Gemini
Unfortunately, I only have access to the ratings and feedback for the second article "Daily Observance of the Spring Festival." To accurately compare and contrast the two articles and provide an overall rating for each, I would need information about the first article "Day 8: Blessings and Harvest Hopes." If you could share the relevant details about the first article, I would be happy to compare them both and provide a comprehensive answer.
However, based on the feedback provided for "Daily Observance of the Spring Festival," it seems to be a well-rounded and informative article with a score of 3.86 (Very Good). It showcases strengths in structure, engagement, and cultural depth, but could benefit from including specific references and going deeper in its analysis.
Without information about the first article, it's impossible to say definitively which one is better. However, the provided feedback can be used to guide further revision and improvement for both pieces.
ChatGPT encountered a few minor issues, most of which were not significant enough to document. On the other hand, Gemini proved to be less capable than it initially seemed.
ChatGPT
ChatGPT failed to provide the overall score for the first time, which was surprising since the prompts for the editor have been consistent for a few days now.
Gemini received a high score, even though it focused on the subject matter of the following day. This could be due to user error, and I will need to check the criteria next round.
Gemini (better, but Bard is leaking out…)
While the eighth day is the eve of the Jade Emperor’s birthday, it's unclear why it mentioned the subject of the ninth day at all. However, I won't research it because of the Bard issue.
When asked to provide the ratings on the article it authored, it claimed to be unable to access the website it cited as a reference. Interestingly, it had 2 additional drafts, both of which succeeded.
It failed to apply the criteria to GPT’s article until asked specifically to do so. This can be chalked up to user error if you like. After specifying it should do so, it seemed to have taxed its memory, so it forgot the ratings for its own article for the follow-up prompt.
"Gemini" is Latin for twins, which leads me to wonder if we got the less capable one today. The AI's engagement with the eighth day revealed both strengths and inconsistencies. ChatGPT's unexpected scoring and Gemini's inability to focus on the requested subject matter, as well as later issues with applying the ratings, suggest that there is room for improvement in my own prompts and Gemini's memory.
Refining AI Evaluation Methods
Refining the approach for these experiments led to a focused comparison of ChatGPT and Gemini's writing capabilities by removing the distraction of image related tasks. This provided clearer observations into each AI's ability to generate written content. This process further highlighted the need for adaptable approaches that could accurately evaluate the nuanced capabilities of AI tools, thereby underscoring the potential use in the workplace.
Key Insights
Positive Observations:
Removing image-related tasks simplified the comparison of writing styles, demonstrating improved writing capabilities despite Gemini veering off-topic.
Encountered Challenges:
Precise prompts that cater to each AI is essential for effectively directing their responses; failure to do so has regularly led to undesirable results.
Despite clear objectives and the provision of supplementary data, Gemini deviated from the specified subject matter and effectively negated the value of an experiment.
Resources
Role Maker v2.4, a GPT for creating professional roles I continue to iterate on.
As an eternal tinkerer, my curiosity, passion, and sheer stubbornness fuel a relentless desire to experiment, learn, and share knowledge, which keeps my creative spirit ignited. I'm constantly looking for new areas to explore, driven by imagination to see where new and evolving technologies might take me.
Driven by passion, not profit, though a coffee is always welcome.
Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of Amazon Web Services (AWS). The author is a UX designer at Amazon Web Services (AWS) and has no involvement in, nor does their work pertain to, any collaborative agreements that AWS may have with Anthropic, the creators of Claude. The insights and analyses presented here are entirely independent and unrelated to any projects or initiatives between AWS and Anthropic. All content in this post is based on publicly available interfaces and is not influenced by the author's employer.