We study whether automatically-induced prompts that effectively extract information from a language model can also be used, out-of-the-box, to probe other language models for the same information. After confirming that discrete prompts
induced with the AutoPrompt algorithm outperform manual and semi-manual
prompts on the slot-filling task, we demonstrate a drop in performance for AutoPrompt prompts learned on a model and tested on another. We introduce a
way to induce prompts by mixing language ...
We study whether automatically-induced prompts that effectively extract information from a language model can also be used, out-of-the-box, to probe other language models for the same information. After confirming that discrete prompts
induced with the AutoPrompt algorithm outperform manual and semi-manual
prompts on the slot-filling task, we demonstrate a drop in performance for AutoPrompt prompts learned on a model and tested on another. We introduce a
way to induce prompts by mixing language models at training time that results
in prompts that generalize well across models. We conduct an extensive analysis
of the induced prompts, finding that the more general prompts include a larger
proportion of existing English words and have a less order-dependent and more
uniform distribution of information across their component tokens. Our work provides preliminary evidence that it’s possible to generate discrete prompts that can
be induced once and used with a number of different models, and gives insights
on the properties characterizing such prompts.
+
