Skip to content

Feat/user summary prompt - #444

Closed
astkwcz wants to merge 1 commit into
nextcloud:mainfrom
astkwcz:feat/user-summary-prompt
Closed

astkwcz wants to merge 1 commit into
nextcloud:mainfrom
astkwcz:feat/user-summary-prompt

Conversation

@astkwcz

@astkwcz astkwcz commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

User defined system prompt for summary task

Related work:

This PR is stacked on #437 :

Feel free to check #437, it describes the whole work for system prompts for different task types. It also provide a description of User defined system prompt for summary and presents UI look. Hierarchy is as follows:

  • user defined system prompt (Complexity and Format options are ignored)
  • admin defined system prompt (Complexity and Format options are appended)
  • default system prompt

@edward-ly asked to move the user-defined system prompt part out of #437 and suggested to add user-defined system prompt part in separate PR with justification.

Justification for user defined system prompts

I agree with @edward-ly that it adds complexity for the user. The user has to know a little about how system prompts work for LLM and how to write instructions. On the other hand, this functionality is available after clicking Show advanced options so we can split users into two group:

  • base options - for everyday users, nothing to configure (by the user) here
  • advanced options - for a little bit more advanced users, who are able to write their own instructions for LLM

Use cases and why admin prompt is not enough

Admin defined system prompt for summary is a good option but it has to be general, because it has to work for everyone.

User-defined system prompt gives a user more control. This control might be required for different type of text to summarize. For example:

  • meeting notes - someone gives a transcript as an input and asks to summarize a meeting and extract decisions, action items etc
  • code - user might want to summarize some log file and call out errors/warnings
  • contracts - user might tell LLM to summarize it and extract parties, duties and deadlines
  • academic papers - user defined system prompt might include instruction to preserve citations, authors and key-findings

So those are field-specific summary tasks types. In general it is a summary, but based on the domain, the user might need different kind of information to be extracted.

What is more, with user defined system prompt for summary a user can set output format themselves for their needs:

  • tabular data
  • markdown format with ## headings
  • company specific structure of a file
    • specific section names
    • specific language
    • some sections bullet points only
    • different sections with paragraph format
    • section that only includes people and their responsibilities
    • section with deadlines

Output formats can be mixed in summary if user defines that.

It all can be achieved with setting a good instruction for a model, but it is field/user specific instruction. Well written admin system prompt can be a good fit for most cases, but for specific kind of field/work user-defined system prompt might give better results if written correctly.

I would really appreciate your thoughts on these use-cases. If you prefer a different approach I am happy to rework this.

@astkwcz astkwcz mentioned this pull request Oct 9, 2026
1 task done

@edward-ly edward-ly left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello, the way I see it, all of these use cases have existing workarounds or can be implemented in simpler ways.

advanced options - for a little bit more advanced users, who are able to write their own instructions for LLM

Why not just use the Assistant chat or the Generate text task instead, providing the "system prompt" and the user prompt in the same field? Separating them wouldn't provide much extra benefit and only confuses all users even more. Plus, overriding the system prompt means users could theoretically provide non-summary prompts in a summary task, which is clearly not the intention.

So those are field-specific summary tasks types. In general it is a summary, but based on the domain, the user might need different kind of information to be extracted.

An LLM might be able to extract these important parts already without the need for additional prompts. But even if it doesn't, you can still use the same workaround as before.

a user can set output format themselves for their needs

I would recommend adding these additional options to the Format input parameter instead. If users require any custom formatting, I would again point them to the same workaround as before.

@astkwcz

astkwcz commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Why not just use the Assistant chat or the Generate text task instead, providing the "system prompt" and the user prompt in the same field?

Considering workaround: it works as long as the input fits a single request. SummaryProvider chunks long input and sends multiple requests if needed, so each chunk stays within max_input_tokens_per_request limit. Chat with AI does not do that.

overriding the system prompt means users could theoretically provide non-summary prompts in a summary task, which is clearly not the intention.

I agree with that. It can be decided by the admin - I can add an admin toggle to enable/disable user-defined prompts if needed.

I would recommend adding these additional options to the Format input parameter instead.

Considering format: I think that adding tables, headings etc format is a good idea. My proposal was about formats that do not fit a fixed enum.

@astkwcz
astkwcz requested a review from edward-ly October 12, 2026 08:01
@github-actions

Copy link
Copy Markdown

Hello there,
Thank you so much for taking the time and effort to create a pull request to our Nextcloud project.

We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process.

Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6

Thank you for contributing to Nextcloud and we hope to hear from you soon!

(If you believe you should not receive this message, you can add yourself to the blocklist.)

@edward-ly

Copy link
Copy Markdown
Contributor

Considering workaround: it works as long as the input fits a single request. SummaryProvider chunks long input and sends multiple requests if needed, so each chunk stays within max_input_tokens_per_request limit. Chat with AI does not do that.

Sure, chunking might not make the most sense for chat, but other tasks such as Improve text do already have chunking built in. You can run a separate task there to instruct the LLM to reformat text as well. What do you think?

I agree with that. It can be decided by the admin - I can add an admin toggle to enable/disable user-defined prompts if needed.

Considering format: I think that adding tables, headings etc format is a good idea. My proposal was about formats that do not fit a fixed enum.

Hm, I think you can use the Improve text task for these cases instead as well. That might be the easiest place to input your custom prompts or formatting. In the meantime, if you have ideas for additional formatting options to add to the enum, feel free to do so.


Also, now that #437 is merged, please rebase the branch, remove the extra commits from that PR, and resolve any additional conflicts too.

Signed-off-by: Arkadiusz Sitkiewicz <a.sitkiewicz@wbgroup.com>
@astkwcz
astkwcz force-pushed the feat/user-summary-prompt branch from deee1ec to 945c8ce Compare October 28, 2026 08:24
@astkwcz

astkwcz commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Improve text do already have chunking built in.
That might be the easiest place to input your custom prompts or formatting.

I wasn't aware of that. In that case I agree that it is valid workaround and we don't need user-system-prompt option.

In the meantime, if you have ideas for additional formatting options to add to the enum, feel free to do so.

Nice to hear that. For now I would not add formatting options just in case. If I run into a different formatting style that is general and I actually use I will open a follow-up PR for it.

Also, now that #437 is merged, please rebase the branch, remove the extra commits from that PR, and resolve any additional conflicts too.

Done ✅

Thanks for collaboration 😊

@edward-ly

Copy link
Copy Markdown
Contributor

I wasn't aware of that. In that case I agree that it is valid workaround and we don't need user-system-prompt option.

Nice to hear that. For now I would not add formatting options just in case. If I run into a different formatting style that is general and I actually use I will open a follow-up PR for it.

Appreciate it, but going back to the original question, then: what other use cases does this PR provide that isn't already covered? I'm not sure if I see a need to merge this at all otherwise.

@astkwcz

astkwcz commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Appreciate it, but going back to the original question, then: what other use cases does this PR provide that isn't already covered? I'm not sure if I see a need to merge this at all otherwise.

Ok, I looked at the code of TextToTextImproveProvider.

Both providers send user's instructions together with a chunk
There is a slightly difference between Improve text and introduced user-system-prompt in SummaryProvider. In summary provider there is a reduction step, whereas Improvde text does not cover that.

Improve text:

  • the input is split into chunks
  • there is a single foreach ($chunks as $chunk) loop
  • the result is the per-chunk LLM outputs simply combined

Summarize:

  • the input is split into chunks
  • there is a do while ($oldNumChunks > $newNumChunks) loop that combines the per-chunk results into one summary
  • this summary is treated as the new input. The input (or summary) is again split into chunks.
  • the mechanism is repeated until loop condition fails

So there is a difference in generated final output:

  • Improve text returns chunked pieces joined together
  • SummaryProvider reduces everything into one coherent summary because of do while loop. It might summarize better for long documents, transcripts etc

The concrete example that I came up with is the case that we have a long transcript of a meeting and want a summarized version of it with fixed sections, for example: 1. Meeting purpose, 2 Attendants. 3. Deadlines.
So those section names are a part of user instructions for LLM.

If the input is split into 3 chunks, the output will be

  • Improve text: the sections repeated 3 times (once per chunk)
  • Summary: one coherent summary with the sections appearing once

That's the case where a user-system-prompt for summary adds something over the Improve text workaround.
I know that I said we don't need user-system-prompt option, but looking at the reduction step I think that there's a still some advantage.
Nevertheless, if you think that is worth the extra option, keep it. If you decide it is not worth it, I am happy to close this PR. Your call.

@edward-ly

Copy link
Copy Markdown
Contributor

There is a slightly difference between Improve text and introduced user-system-prompt in SummaryProvider. In summary provider there is a reduction step, whereas Improvde text does not cover that.

Hm, that is a fair point, although in that case, I would consider that a bug of the Improve text task for which you can file a new issue where we can discuss ways to improve that.

As for the summary task itself, if the main use case for the custom system prompt would be to input a custom format, then I would make it clearer that the option should be used as such. So, instead of labeling the box Summary system prompt (which itself might be too technical for users anyway), it should be something like Custom format description or Describe how the assistant should format the text. Would that be sufficient?

And regarding your example of 1. Meeting purpose, 2 Attendants. 3. Deadlines, the call summary bot can provide an agenda, attendance, and next tasks already, so I don't see a need to re-implement the same thing here. For deadlines, that could be a new feature request for which you can file a new issue there.

@astkwcz

astkwcz commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

I would consider that a bug of the Improve text task for which you can file a new issue where we can discuss ways to improve that.

I am not sure if it is a bug. Improve does not mean reduce, whereas summarize always means reduce.
There is also the same mechanism foreach ($chunks as $chunk) in TranslateService.php‎. It makes sense because in case of translation we do not want to reduce text, input and output lengths are nearly equal. I think there is the same logic for Improve text.

As for the summary task itself, if the main use case for the custom system prompt would be to input a custom format, then I would make it clearer that the option should be used as such. So, instead of labeling the box Summary system prompt (which itself might be too technical for users anyway), it should be something like Custom format description or Describe how the assistant should format the text. Would that be sufficient?

Ok, I agree that Summary system prompt might be too technical and I can rename it. I also suggest broader wording, because the instructions are not limited to formatting, a user might also want to change what the summary covers - skip or group information, focus on certain parts etc. I suggest:

  • label:Custom summary instructions
  • helper text such as "Write your own instructions for the summary. When set, the Format and Complexity options are ignored"

What do you think about that?

And regarding your example of 1. Meeting purpose, 2 Attendants. 3. Deadlines

That was just an example to illustrate how chunking differs between those two tasks - reduction of a text vs keeping nearly the same text length. I did not mean to re-implement the call summary bot here. I agree that would be out of scope

@edward-ly

Copy link
Copy Markdown
Contributor

Alright, after some internal discussion, we decided to close this PR, mainly because it is not possible to limit the scope of this new prompt option so that it can only be used for actual summarization, which is the only thing that the "summarize" task should actually do (some of the use cases you provided fall outside of that scope). Not to mention, having this user prompt override or conflict with other options might not be the best design decision either (our worry is that it can create extra confusion and complexity for not enough gain).

As an alternative, though, we would suggest creating a new task type similar to the "Improve text" task that can handle those other use cases. Naming it something like "Reduce text", this task could be a more general task that can condense text in more ways outside of just summarization (for example, extracting specific parts of the text specified by the user) while getting around the technical limitations of the "Improve text" task. This will require additional changes outside of the integration_openai app too, but if you're up for it, we'd be happy to take a look at your code again at that point!

@edward-ly edward-ly closed this Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants