A Perplexity essay
(This is what perplexity says about herself: How is Perplexity AI different?)
Formulating the task: The question
I spent about 3 hours formulating the task below (I could not help but talk politely to it; like talking to a human being).
Could you please write a scientific essay, a research paper in a style and of size - the shorter, the better - that is acceptable for an internationally recognized, peer-reviewed periodical about the "Tree banks usage in linguistics" focusing on the Universal Dependencies (UD) approach? Cover these topics
Typical use of linguistic corpora in:
Theoretical linguistics
Experimental linguistics
Philology
Cultural anthropology
Education (provide links to resources; printed books or articles)
Second language acquisition (provide links to resources; printed books or articles)
History of the UD approach
Theoretical foundations of the UD
Current status of the project launched by the UD group
Corpora implementations by the third parties using the UD approach
Pros and Cons for using the UD for building tree banks
Review other successful approaches to building tree banks
Review and comment on the research papers, written on the topics of the listed in the bullet-item 1 above from the point 1.a) to 1.f). It would be nice getting some quantitative analysis, for example,
How many research papers are written per the areas listed in the bullet-items 1a - 1f in the last 12 years?
How many of those are highly cited?
Give an overview of the research that is done in the top 6 highly-cited papers that you reviewed (see bullet-item #8 above). Provide a measure of success, something like a 1) major breakthrough discovery, 2) a discovery in a particular area, 3) new view on a known problem, 4) confirmation/ refutation of current theory, 5) no good at all, etc.
The answer
The Perplexity generated the answer in about 40 sec.
Before analyzing the answer I would like to direct your attention to the structure of the response page:
Before the generated answer content there is a section with Sources. Then the answer text comes.
On the right-top corner you can see relevant images. Please note Search videos, Generated images sections below the images.
At the bottom of the response you can find a list of recommended follow up questions Related to your inquiry.
Now to the essence of the response. Note that the references are numbers linked to words with no separators; typically, in scientific literature they put consecutive reference number in square brackets like this: … syntax [2]:
Note the title - it is not the one that I provided
The link to the reference in … cognitive representations of syntax2 is broken.
The reference at the end of The UD framework facilitates diachronic comparisons within and across languages3 looks irrelevant - it is about language acquisition by “Mowglis”.
The reference in … an underutilized application4 has nothing to do with Cultural anthropology (or corpora, tree banks, UD). I am spending time on this because it looks like a good review. It is a review on how the Distributional semantics approach is used to advance our knowledge of language. BTW, for detailed (layman’s level) description Distributional semantics, how such distribution is calculated, stored, and used see section 23.2.5 Ուսանում. The referenced review outlines 4 “ways for distributional semantic research to contribute to linguistic theories” (it would have been more appropriate reference for the Theoretical linguistics section):
Exploratory. Distributional data such as similarity scores and nearest neighbors can be used to explore data on a large scale.
A tool to identify instances of specific linguistic phenomena. For instance, changes in distributional representations of words across time can be used to systematically harvest potential instances of semantic change in diachronic data.
A test bed for linguistic hypotheses, by testing predictions in distributional terms.
The actual discovery of linguistic phenomena or theoretically relevant trends in data.
The references 5 and 6 are relevant, but too generic: the former points to the UD toolkit and the latter - to the Cambridge University press webpage on Studies in Second Language Acquisition
The data in the Research Trends Analysis section looks impressive and does not go against common sense, however, it is hard to evaluate the accuracy of “bibliometric analysis” (wait for a surprise - see my follow up question: Please clarify the methodology and list the sources of information you have used for writing the Research Trends Analysis section. and note a confession in hallucinating).
The Top Cited Papers Overview looks impressive too. However, it is not what I asked for.
I assume (?) the link in the conclusions section is supposed to illustrate how to balance “universality with language-specific nuances.”
Impressions
To tell the truth I expected a better answer, but after thinking a bit I understood that it is not that bad. I fell into the trap that I joke about. When people say that “AI is stupid”, I tell them this [old, pre-GPT] joke:
A man plays chess with his dog on the beach. Bypassers stop and watch in awe. One of them tells the man: “This is unbelievable. Your dog is incredibly smart”. The man replies with smirk: “No, it’s not, I am winning 7 to 1”.
Now. Think about the Perplexity’s reply: a piece of metal, which learned how to speak (write) by “reading” a lot of texts, writes in perfect English that even native speakers admire. Maybe a sophomore can write a better, deeper, more accurate essay, but just imagine the time s/he needs to spend. But you will probably admit that not all graduates, even post graduate students, can write a better essay. Just think which of your students can write an essay of this complexity. Even considering that you formulated the task - I am not even talking about planning the essay.
Note also the quality of the follow up questions suggested in the Related section.
How I use the Perplexity.ai
I use Perplexity in 2 major areas: 1) when I need a quick answer to a question (because Google returns tons of links that I do not have time to go through) and 2) as an assistant programmer or IT consultant. The second is the most efficient use.
I typically ask to write a program:
That needs knowledge of third party (or even Java) libraries that I have no ide about. The most recent and successful experience was with image processing functionality.
Write UnitTests for programs written by Perplexity or me.
A stand-by assistant to answer my "stupid" questions, if I forgot or never knew simple things in programming.
The above 2 cases look reasonable. I continued writing code that I know how to write. But why? Recently I realized that if I know how to implement some functionality does not mean that I need to spend time typing tons of code via keyboard. Why not delegate this to Perplexity too? After all, when I was working on a project for eBay, I saw a couple of presentations on the gitHub Copilot usages.
There are 2 types of integrations with the IDE - one that writes the code directly into your IDE editor window and the other - in a pop-up. The difference is to me a superficial convenience. In the first case you come to a point and tell the Copilot to continue coding; in the pop-up mode you tell Copilot to write such and such code and then copy and paste it wherever you need it.
Understanding that Perplexity is not trained specifically for programming I decided to try anyway. I do not regret it.
These are some observations:
It is very useful for simple tasks that you can clearly define. It gives the answer - typically 100% working code - in seconds. This is most useful in cases where the functionality can be simply defined, but you need to spend a lot of time researching third party libraries.
Sometimes it makes human mistakes in the code: it scares me and gives me chills.
It does not handle the requests, which contain too many [mini] tasks/subrequests, well. Like my task to write a scientific essay. I continued refining the essay by reasking subrequests. See follow up questions in the same place.
As you can see I got a confession (-:) from the machine: it lied to me. You need to press harder to get the results.
In coding (programming) I point out errors and ask Perplexity to fix them. Sometimes it takes several iterations.
I continued resubmitting parts of my original request in hope to get better answers.
After pressing more I got another confession about the Top Cited Papers Overview.
Conclusions
The essay ended up being very shallow. I did not get an answer to my main question: How linguistic corpora, precisely tree banks, are used for groundbreaking research?
Several relevant references were placed incorrectly. The cases of confident lies and misinformation are not pleasant (but have been well documented). By double checking, reformulating, and retrying requests you can get useful information, if it is not a completely accurate, satisfactory answer. This prompt engineering skill everyone should master. It becomes a separate area of research and development. And linguists might contribute a lot. [I'll continue pushing the envelope until I convince interested parties in Polytechnic and Linguistics University to start delivering prompt engineering courses. I am sure high school is even better place to start.]
I recommend using Perplexity for initial research, simple coding, writing UnitTest, optimizing code (did not try yet), writing simple essays, writing emails, translating into Armenian, etc.
The so called LLMs become better and better on a daily basis.
I that the user should adjust to the tools and use them where they work better. Note that people get pets not for playing chess. Everyone can find aspects of LLM use and become skilful in communicating with them. It might be not easy, but adjusting to use computers, internet, smartphones, was not easy either. For linguists it might play a role of Python language or RegEx tutor. At very list as a stand-by - day and night - assistant.
It is important timely introduction of LLM (other technologies) into the cultural fabric of nation. Otherwise, the nation will never catch up, like it happened in Soviet Union with Cybernetics and Genetics (Molecular Biology).
Today we cannot imagine our lives without computer, web, smartphone, etc. In 2-3 years you will add LLMs to the list. The AI Singularity is expected in 2029. Get ready.
P.S. Note, that I used free access. Paid services - I suppose - are much better.
Comments